Exception when reading million characters in JSON file [OutOfMemoryException]

Question

I have downloaded a JSON file recorded from Azure Blob Storage. The file size is 137MB.

The character and line properties when open using Notepad++ as per below image:

It takes around 1-2 seconds when I choose "Edit with Notepad++" from file context menu. So, I'm decided to create a program to make a JSON converter to CSV file format. But seems, I have faced some kind of exception. Currently, for viewing the JSON content, I will show in RichTextBox as it can view the content before I decide to convert to CSV.

Event to start load:-

private async void txtjsonname_DoubleClick(object sender, EventArgs e)
{
    OpenFileDialog ofd = new OpenFileDialog();
    ofd.Filter = "JSON Files (*.json)|*.json";
    ofd.InitialDirectory = @"C:\";
    ofd.Title = "Select single json file to be converted";
    ofd.Multiselect = false;
    if (ofd.ShowDialog() == DialogResult.OK)
    {
        rtbstat.Text = null;
        txtcsvname.Text = null;
        txtjsonname.Text = null;
        lblcsvpath.Text = null;
        lbljsonpath.Text = null;
        rtbjson.Clear();
        txtjsonname.Text = Path.GetFileName(ofd.FileName);
        lbljsonpath.Text = Path.GetDirectoryName(ofd.FileName);

        if (await LoadJSONtoRTB(ofd.FileName))
        {
            rtbjson.WordWrap = false;
            rtbstat.Text = "Load file finished! " + (rtbjson.Lines.Count()).ToString() + " line(s) detected | " + rtbjson.Text.Length.ToString() + " character(s) detected";
            txtcsvname.Text = Path.GetFileNameWithoutExtension(ofd.FileName) + ".csv";
        }
    }
    await Task.Delay(1000);
}

The code that I have try and face exception:-

First Approach: First code:

private async Task<bool> LoadJSONtoRTB(string path)
    {
        try
        {
            foreach (var line in File.ReadLines(path))
            {
                rtbjson.Text = line;
            }
            await Task.Delay(10);
            return true;
        }
        catch (Exception)
        {
            return false;
        }
    }

Second code:

private async Task<bool> LoadJSONtoRTB(string path)
    {
        try
        {
            using (FileStream fs = File.Open(path, FileMode.Open, FileAccess.Read, FileShare.ReadWrite))
            using (BufferedStream bs = new BufferedStream(fs))
            using (StreamReader sr = new StreamReader(bs))
            {
                string line;
                while ((line = sr.ReadLine()) != null)
                {
                    rtbjson.AppendText(line);
                }
            }
            await Task.Delay(10);
            return true;
        }
        catch (Exception)
        {
            return false;
        }
    }

Exception: An unhandled exception of type 'System.AccessViolationException' occurred in System.Windows.Forms.dll

Additional information: Attempted to read or write protected memory. This is often an indication that other memory is corrupt.

Second Approach:-

private async Task<bool> LoadJSONtoRTB(string path)
{
    try
    {
        StreamReader sr = new StreamReader(@path);
        while (!sr.EndOfStream)
            rtbjson.Text += sr.ReadLine();
        await Task.Delay(10);
        return true;
    }
    catch (Exception)
    {
        return false;
    }
}

Using above code, it running about 12 minutes at when I'm put the breakpoint to see the progress.

12 minutes with 6million length read.

Is there any ways to show the text file (json/txt) with example of 64 millions characters length like notepad++ only takes 1-2 seconds to view the file?

Operations like this: `rtbjson.Text += sr.ReadLine();` on strings are very slow for a huge amount of text, because on each step it creates new variable for result of this operation. Instead try to user StringBuilder and AppendText method and let me knwo how it works — Marcin, Jan 20 '17 at 09:30
First approach is likely better than second. What if you disable graphical updates while appending the lines? See [How do you prevent a RichTextBox from refreshing its display?](http://stackoverflow.com/a/26088349/3744182). — dbc, Jan 20 '17 at 10:07
Also, `RichTextBox.LoadFile()` might help. See [C# - Read in a large (150MB) text file into a Rich Text Box](http://stackoverflow.com/a/9498508/3744182). — dbc, Jan 20 '17 at 10:11
Standard Windows controls are designed for normal amount of text. Editor that support very large amount of text are specifically designed for that purpose. — Phil1970, Jan 20 '17 at 14:22

Matteo Umili · Accepted Answer · 2017-01-20T10:45:12.517

2

I suspect that Notepad++ loads the entire file in memory with something equivalent to System.IO.File.ReadAllText. Also, there is no benefit in appending every row of the file to the string, the final result is the same memory occupied. With RichTextBox the best you can do is:

richTextBox1.Text = System.IO.File.ReadAllText(filePath);

Anyway, Notepad++ uses Scintilla that is faster than RichTextBox.

You can try using ScintillaNET that is a wrapper over Scintilla.

You can set the control text in the same way you do with RichTextBox:

scintilla1.Text = System.IO.File.ReadAllText(filePath);

edited Jan 20 '17 at 10:45

answered Jan 20 '17 at 10:12

Matteo Umili

3,412
1
19
31

I have try exploring this Scintilla..and it was superb. I'm currently loading 88,940,748 length in 7.118 seconds !!!!! Now, I need to search for reading for 143,685,300 length. Currently, it just return blank after ReadAllText. My 5! – Luiey Jan 21 '17 at 02:44
Currently reading 475,212,842 length it takes 6seconds to load. The problem when loading big size file are the Build architecture. I change from AnyCPU to x64 on Build Platform Target. All solved. – Luiey Jan 23 '17 at 06:34

lobiZoli · Answer 2 · 2017-01-20T12:15:30.070

Your LoadJSONtoRTB method runs asynchronously. Hence you're trying to update the gui (textbox) from the wrong thread. This approach will help you run the gui update on the right thread:

this.Invoke(new Action(() => { rtbjson.Text += sr.ReadLine(); }));

Of course there are more efficient methods populating the control with large amount of text like StringBuilder. The important take away is to always update the gui on the gui thread. And that can be done by running Form.Invoke

Exception when reading million characters in JSON file [OutOfMemoryException]

2 Answers2