Optimizing List

Question

I am a little new to C# and I'm having performance issues with this. In my program, people import a .txt list and the program makes a list out of it; the problem is its consuming too much RAM, crashing PC's with low memory. I thought of using 'yield' without success. Any ideas?

private List<string> ImportList()
{
    try
    {
        using (var ofd = new OpenFileDialog() { Filter = "Text files (*.txt) | *.txt" })
        {
            if (ofd.ShowDialog() == DialogResult.OK)
            {
               return File.ReadAllLines(ofd.FileName).ToList();
            }
        }

        return null;
    }
    catch(OutOfMemoryException ex)
    {
        MessageBox.Show("The list is too large. Try using a smaller list or dividing it.", "Warning!");
        return null;

    }
}

1. OutOfMemoryException or 2. if set to x64 and , the app will consume all the PC memory. — Alberto Lopez Rodriguez, Nov 24 '15 at 04:39
You've said that it hurts when you do that. So *don't do that*. The file is sitting there perfectly happy on disk; why are you reading it entirely into memory in the first place? — Eric Lippert, Nov 24 '15 at 04:40
A 550mb file is nothing assuming its just the one file. There must be another reason. — , Nov 24 '15 at 04:40
I thought ReadAllLines didnt load it on the memory, just read from the disk. What should I do then? — Alberto Lopez Rodriguez, Nov 24 '15 at 04:53
I would strongly recommend you to take a look at this post. The answer might be the very answer you are looking for http://stackoverflow.com/questions/2009217/whats-the-max-items-in-a-listt — Daneau, Nov 24 '15 at 04:59
you mustn't get all data into memory if you don't enough memory. And you should give more information, for example what do you want with this huge file? have you looked at memory mapped file too ? — RockOnGom, Nov 24 '15 at 05:06

score 5 · Accepted Answer · edited Feb 10 '18 at 12:41

the method ReadlAllLines returns an array of string, not a List => File.ReadAllLines Method (String)

I think that you shuld use ReadLines(), check this Question about the diferences between ReadLines and ReadlAllLines:

is there any performance difference related to these methods? YES there is a difference

File.ReadAllLines() method reads the whole file at a time and returns the string[] array, so it takes time while working with large size of files and not recommended as user has to wait untill the whole array is returned.

File.ReadLines() returns an IEnumerable and it does not read the whole file at one go, so it is really a better option when working with large size files.

From MSDN:

The ReadLines and ReadAllLines methods differ as follows:

When you use ReadLines, you can start enumerating the collection of strings before the whole collection is returned; when you use ReadAllLines, you must wait for the whole array of strings be returned before you can access the array. Therefore, when you are working with very large files, ReadLines can be more efficient. Example 1: File.ReadAllLines()
string[] lines = File.ReadAllLines("C:\\mytxt.txt");
Example 2: File.ReadLines()
foreach (var line in File.ReadLines("C:\\mytxt.txt"))
{

   //Do something     

}

Response for Sudhakar Tillapudi

Arguably should have marked question as _duplicate_ rather than quoting another answer — , Nov 24 '15 at 05:07
Maybe Micky, but the question was "Optimizing List" not diference between ReadALlLines and ReadLines — Bradley, Nov 25 '15 at 16:58

score 0 · Answer 2 · edited May 23 '17 at 11:59

Read Big TXT File, Out of Memory Exception

I just copied paste the solution from other question. See if it works.

foreach (var line in File.ReadLines(_filePath))
{
    //Don't put "line" into a list or collection.
    //Just make your processing on it.
}

ReadLines returns IEnumerable<string>.File.ReadLine

Concept is to not load all lines into a list at once. Even if you want to process them, process them line by line using IEnumerable instead of List.

score 0 · Answer 3 · answered Nov 24 '15 at 05:09

If the exception occurs at ReadAllLines, try this:

Use a StreamReader to read the file line by line and ad it to the list. Something like this:

using (StreamReader sr = new StreamReader (ofd.FileName)) {
    while (!sr.EndOfStream) {
        yourList.Add (sr.ReadLine());
    }
}

If the exception occurs at ToList, try this:

You should get the array returned by ReadAllLines first, and use a foreach loop to add the array elements to the list.

foreach (var str in arrayReturned) {
    yourList.Add (str);
}

If this still does not work, use the ReadLines method in the same class. The difference between ReaDAllLines and ReadLines is that the latter returns an IEnumerable<string> instead of a string[]. An IEnumerable<string> use deferred execution. It will only give you one element when you ask it to. Jon Skeet's book, C# In Depth talks about this in detail.

Here is the docs for ReadLines for more information:

https://msdn.microsoft.com/en-us/library/dd383503(v=vs.110).aspx

Optimizing List

3 Answers3