Out Of Memory when reading big files

Question

I am creating a tool which analyze the data quality of a file. So i need to read each line of the file and analyze each one of them. I also need to store in memory all the lines of my file as the user will be able to drill down to specific sections. So basically all works fine for a file containing thousands of rows. However when trying with a CSV file containing more than 4 millions rows i m getting an out of memory exception. I thought C# would be able to handle a few millions of data in its memory cache but does not seem like it. So i am a bit stuck and dont know what to do. Maybe my piece of code is not the most performant so if you can tell me a way to improve it that would be great? Just to keep in mind that i need to have all lines of the file in memory because depending of the user's action i need to access speciifc lines to display them to the user.

Below is the call which reads every single line

using (FileStream fs = File.Open(this.dlgInput.FileName.ToString(),   FileMode.Open, FileAccess.Read, FileShare.Read))
using (BufferedStream bs = new BufferedStream(fs))
using (System.IO.StreamReader sr = new  StreamReader(this.dlgInput.FileName.ToString(), Encoding.Default, false, 8192))
{
    string line;
    if (this.chkSkipHeader.Checked)
    {
        sr.ReadLine();
    }

    progressBar1.Visible = true;
    int nbOfLines = File.ReadLines(this.dlgInput.FileName.ToString()).Count();
    progressBar1.Maximum = nbOfLines;

    this.lines = new string[nbOfLines][];
    this.patternedLines = new string[nbOfLines][];
    for (int i = 0; i < nbOfLines; i++)
    {
        this.lines[i] = new string[this.dgvFields.Rows.Count];
        this.patternedLines[i] = new string[this.dgvFields.Rows.Count];
    }

    // Read and display lines from the file until the end of 
    // the file is reached.
    while ((line = sr.ReadLine()) != null)
    {
        this.recordCount += 1;
        char[] c = new char[1] { ',' };
        System.Text.RegularExpressions.Regex CSVParser = new System.Text.RegularExpressions.Regex(",(?=(?:[^\"]*\"[^\"]*\")*(?![^\"]*\"))");
        String[] fields = CSVParser.Split(line);
        ParseLine(fields);
        this.lines[recordCount - 1] = fields;
        progressBar1.PerformStep();
    }
}

And below is the ParseLine function which also keeps in memory via arrays some analysis needed:

private void ParseLine(String[] fields2)
{
    for (int j = 0; j <= fields2.Length - 1; j++)
    {
        if ((int)this.dgvFields.Rows[j].Cells["colSelected"].Value == 1)
        {
            /*' ************************************************
            ' Save Number of Counts by  Value
            ' ************************************************/

            if (this.values[j].ContainsKey(fields2[j]))
            {
                //values[0] = Dictionary<"TEST", 1> (fields2[0 which is source code] = count])
                this.values[j][fields2[j]] += 1;
            }
            else
            {
                this.values[j].Add(fields2[j], 1);
            }

            /* ' ************************************************
            ' Save Pattern Values/Counts
            ' ************************************************/

            string tmp = System.Text.RegularExpressions.Regex.Replace(fields2[j], "\\p{Lu}", "X");
            tmp = System.Text.RegularExpressions.Regex.Replace(tmp, "\\p{Ll}", "x");
            tmp = System.Text.RegularExpressions.Regex.Replace(tmp, "[0-9]", "0");


            if (this.patterns[j].ContainsKey(tmp))
            {
                this.patterns[j][tmp] += 1;
            }
            else
            {
                this.patterns[j].Add(tmp, 1);
            }

            this.patternedLines[this.recordCount - 1][j] = tmp;
            /* ' ************************************************
             ' Count Blanks/Alpha/Numeric/Phone/Other
             ' ************************************************/


            if (String.IsNullOrWhiteSpace(fields2[j]))
            {
                this.blanks[j] += 1;
            }
            else if (System.Text.RegularExpressions.Regex.IsMatch(fields2[j], "^[0-9]+$"))
            {
                this.numeric[j] += 1;
            }
            else if (System.Text.RegularExpressions.Regex.IsMatch(fields2[j].ToUpper().Replace("EXTENSION", "").Replace("EXT", "").Replace("X", ""), "^[0-9()\\- ]+$"))
            {
                this.phone[j] += 1;
            }
            else if (System.Text.RegularExpressions.Regex.IsMatch(fields2[j], "^[a-zA-Z ]+$"))
            {
                this.alpha[j] += 1;
            }
            else
            {
                this.other[j] += 1;
            }

            if (this.recordCount == 1)
            {
                this.high[j] = fields2[j];
                this.low[j] = fields2[j];
            }
            else
            {
                if (fields2[j].CompareTo(this.high[j]) > 0)
                {
                    this.high[j] = fields2[j];
                }

                if (fields2[j].CompareTo(this.low[j]) < 0)
                {
                    this.low[j] = fields2[j];
                }
            }
        }
    }
}

UPDATE: new piece of code

int nbOfLines = File.ReadLines(this.dlgInput.FileName.ToString()).Count();
        //Read file

        using (System.IO.StreamReader sr = new StreamReader(this.dlgInput.FileName.ToString(), Encoding.Default, false, 8192))
        {
            string line;
            if (this.chkSkipHeader.Checked)
            { sr.ReadLine(); }
            progressBar1.Visible = true;

            progressBar1.Maximum = nbOfLines;
            this.lines = new string[nbOfLines][];
            this.patternedLines = new string[nbOfLines][];
            for (int i = 0; i < nbOfLines; i++)
            {
                this.lines[i] = new string[this.dgvFields.Rows.Count];
                this.patternedLines[i] = new string[this.dgvFields.Rows.Count];
            }

            // Read and display lines from the file until the end of 
            // the file is reached.
            while ((line = sr.ReadLine()) != null)
            {
                this.recordCount += 1;
                char[] c = new char[1] { ',' };
                System.Text.RegularExpressions.Regex CSVParser = new System.Text.RegularExpressions.Regex(",(?=(?:[^\"]*\"[^\"]*\")*(?![^\"]*\"))");
                String[] fields = CSVParser.Split(line);
                ParseLine(fields);
                this.lines[recordCount - 1] = fields;
                progressBar1.PerformStep();
            }
        }

Answer 1

C# has a limit on how big single objects can be (hence the exception). Consider the fact that, even if every string in your array was 1 byte, 4 million bytes would still be around 4 gigabytes and, as I understand it, the default max size of a single object in .NET is 2 gigabytes. This is true regardless of how much memory you have in your system overall.

There are a couple of articles available on Stack Overflow how to create big arrays: I need very big array length(size) in C# and OutOfMemoryException on declaration of Large Array

As I understand it, this is partially the result of how the .NET framework managed the transition from 32 bits to 64 bits. (Note, too, that 2 gigabytes roughly corresponds to the max value of a 32-bit signed int). In newer versions of .NET (after 4.5 according to what I've read but I've never tried it) I guess you can change the max object size to some degree. There are also special classes you can use (e.g. a custom BigArray class) to get around the space limit.

Keep in mind, too, that an array requires that it be able to allocate consecutive memory addresses (that's why you can do constant-time accesses by index - the address is a constant offset from the pointer to the first item, so the framework can figure out the memory location by multiplying the index by 32 or some other constant depending on memory size and adding it to the address in the pointer to the first item to figure out where the item will be). Thus, fragmentation in memory can reduce the effective amount of memory available for arrays.

Answer 2

You need to create helper class which will cache starting positions of each line in entire file.

 int[] cacheLineStartPos;

 public string GetLine (int lineNumber)
 {
     int linePositionInFile = cacheLineStartPos[lineNumber];

     reader.Position = linePositionInFile;

     return reader.ReadLine();
 }

Of course it just an example, and logic can be more complex.

Answer 3

If you need to process a larger amount of data, consider using databases. They were designed for exactly this kind of purpose. You can also query them afterwards with specific request. Probably a key-value store is sufficient for you already. Take a look at https://ravendb.net/ or https://www.mongodb.com/

Answer 4

You should not have all your lines in memeory even if you need the user to act on all data, something similar to baretail, you have to read the lines from disk, and just for the window which is visible to the user, when he seeks more data, then you will stream more from disk with the same window width, but never for all lines, think of files like 40 GB ... it will not be practical to have all of them loaded. Here is an example for how to do that, and as requested from other memebers, here is the code from the answer mentioned, credit to @James King

//  This really needs to be a member-level variable;
private static readonly object fsLock = new object();

//  Instantiate this in a static constructor or initialize() method
private static FileStream fs = new FileStream("myFile.txt", FileMode.Open);


public string ReadFile(int fileOffset) {

    byte[] buffer = new byte[bufferSize];

    int arrayOffset = 0;

    lock (fsLock) {
        fs.Seek(fileOffset, SeekOrigin.Begin);

        int numBytesRead = fs.Read(bytes, arrayOffset , bufferSize);

        //  Typically used if you're in a loop, reading blocks at a time
        arrayOffset += numBytesRead;
    }

    // Do what you want to the byte array and return it

}