1

The files at max could be 1GB

came across these 2 reads

IEnumerable

and

How to: Read Text from a File

Which one is better in terms of efficiency - memory/cpu

Also, where can i read up on what stream reader implements under the hood??

iciw
  • 13
  • 2

1 Answers1

5

Neither - if you're using .NET 4, use File.ReadLines.

If you're using .NET 3.5 or earlier, you can write this yourself more easily than the VB page showed, as VB doesn't have iterator blocks yet (it's getting them in the next version, and in a more flexible way than in C#):

// Optional: create an overload of this taking an encoding
public IEnumerable<string> ReadLines(string file)
{
    using (TextReader reader = File.OpenText(file))
    {
        string line;
        while ((line = reader.ReadLine()) != null)
        {
            yield return line;
        }
    }
}

This is effectively what File.ReadLines does for you anyway.

Then you can use:

foreach (string line in ReadLines("file.txt"))
{
    // Use a line at a time
}

No more than a line or so (the size of the buffer) will be in memory at a time, and it'll be reasonably CPU-efficient too.

All of these will use StreamReader under the hood, which is right and proper as it's the canonical way of converting a stream of binary data to a TextReader. Basically it reads binary data from a Stream and uses an Encoding to convert that data into text as and when it's requested. What more do you need to know?

Jon Skeet
  • 1,421,763
  • 867
  • 9,128
  • 9,194
  • IF they all use stream reader under the hood then isn't the MSDN article of using streamreader the best way to go about it instead of using implementations that use streamreader? – iciw Aug 10 '11 at 21:25
  • What's the rational behind "neither"? (Particular, "in terms of efficiency - memory/cpu".) –  Aug 10 '11 at 21:26
  • 1
    @pst: Why write code that's been written for you already as part of the framework? What makes you think either of the other options would be more efficient than `File.ReadLines`? – Jon Skeet Aug 10 '11 at 21:26
  • @iciw: It's not clear which of the MSDN articles you mean, but the nice thing about getting an `IEnumerable` is that it separates the code which is responsible for "reading a file line by line" (which is completely reusable) from the code which wants to *use* those lines (which is likely to be more specific). – Jon Skeet Aug 10 '11 at 21:28
  • @Jon Skeet: I mean this article http://msdn.microsoft.com/en-us/library/db5x7c0d.aspx – iciw Aug 10 '11 at 21:31