Out of memory exception reading and writing text file

Question

I get an out of memory exception a few seconds after I execute the following code. It doesn't write anything before the exception is thrown. The text file is about half a gigabyte in size. The text file I'm writing too will end up being about 3/4 of a gigabyte. Are there any tricks to navigate around this exception? I assume it's because the text file is too large.

public static void ToCSV(string fileWRITE, string fileREAD)
{
    StreamWriter commas = new StreamWriter(fileWRITE);
    var readfile = File.ReadAllLines(fileREAD);


    foreach (string y in readfile)
    {

        string q = (y.Substring(0,15)+","+y.Substring(15,1)+","+y.Substring(16,6)+","+y.Substring(22,6)+ ",NULL,NULL,NULL,NULL");
        commas.WriteLine(q);
    }

    commas.Close();
}

I have changed my code to the following yet I still get the same excpetion?

public static void ToCSV(string fileWRITE, string fileREAD)
{
    StreamWriter commas = new StreamWriter(fileWRITE);

    using (FileStream fs = File.Open(fileREAD, FileMode.Open, FileAccess.Read, FileShare.ReadWrite))
    using (BufferedStream bs = new BufferedStream(fs))
    using (StreamReader sr = new StreamReader(bs))
    {
        string y;
        while ((y = sr.ReadLine()) != null)
        {
            string q = (y.Substring(0, 15) + "," + y.Substring(15, 1) + "," + y.Substring(16, 6) + "," + y.Substring(22, 6) + ",NULL,NULL,NULL,NULL");
            commas.WriteLine(q);
        }
    }

    commas.Close();
}

You are still doing the same mistake on your changed code, by putting the `FileStream` into a `BufferedStream`. Don't do it. Just open the file with `StreamReader` and read line by line. — Tolga Evcimen, Apr 30 '14 at 07:32

score 1 · Answer 1 · edited May 23 '17 at 12:22

1

In the following post you can find numerous methods for reading and writing large files. Reading large text files with streams in C#

Basically you just need to read the bytes in a buffer which will be reused. This way you will load a very small amount of the file into memory.

edited May 23 '17 at 12:22

Community

1
1

answered Apr 30 '14 at 06:54

Georgi-it

3,676
1
20
23

score 1 · Answer 2 · answered Apr 30 '14 at 06:55

Instead of reading the whole file, try to read and process line by line. That way you do not risk to run into an out of memory exception. Because even if you succeed in organising more memory for the program, one day will come where the file will again be too large.

But the program may loose speed if you use less memory, so basically you'd have to balance the memory use and the execution time. One workaround would be to use a buffered output, reading more than one line at a time or transforming the strings in multiple threads.

score 1 · Accepted Answer · edited Nov 25 '19 at 14:10

Read the file line by line, it will help you to avoid OutOfMemoryException. And I personnaly prefer using using's to handle streams. It makes sure that the file is closed in case of an exception.

public static void ToCSV(string fileWRITE, string fileREAD)
{
    using(var commas = new StreamWriter(fileWRITE))
    using(var file = new StreamReader("yourFile.txt"))
    {
        var line = file.ReadLine();

        while( line != null )
        { 
            string q = (y.Substring(0,15)+","+y.Substring(15,1)+","+y.Substring(16,6)+","+y.Substring(22,6)+ ",NULL,NULL,NULL,NULL");
            commas.WriteLine(q);
            line = file.ReadLine();
        }
    } 
}

Out of memory exception reading and writing text file

3 Answers3

Linked