OutOfMemoryException when reading a string

Question

I have a C# .NET system that takes a JSON data feed and converts it to an object using the Newtonsoft.Json.JsonConvert.DeserializeObject converter.

This process works perfect as long as the JSON string is below a certain size (a few Mb) but as soon as the returned data is large (almost 100Mb) I get the error OutOfMemoryException

This code works great for small data:

// WebClient ------------------------------------------------------------------
var _client = new System.Net.WebClient();
var _content = _client.DownloadString(_url);

but blows up on the last line (DownloadString)

I tried changing to this which also works for small data but it still blew up on the ReadToEnd line when the data grew in size.

using (var _response = (System.Net.HttpWebResponse)_request.GetResponse())
{
    using (System.IO.Stream _dataStream = _response.GetResponseStream())
    {
        using (System.IO.StreamReader _streamReader = new System.IO.StreamReader(_dataStream))
        {
            string _responseFromServer = _streamReader.ReadToEnd();
        }
    }
}

Finally I tried this which worked:

StringBuilder _stringBuilder = new StringBuilder();
using (var _response = (System.Net.HttpWebResponse)_request.GetResponse())
{
    using (System.IO.Stream _dataStream = _response.GetResponseStream())
    {
        using (System.IO.StreamReader _streamReader = new System.IO.StreamReader(_dataStream))
        {
            while (!streamReader.EndOfStream)
            {
                char[] _buffer = new char[4096];
                _streamReader.ReadBlock(_buffer, 0, _buffer.Length);
                var _bufferString = new String(_buffer);
                _stringBuilder.Append(_bufferString);
            }
        }
    }
}

But it blew up with an OutOfMemoryException error when it got to the next line here:

var _results = Newtonsoft.Json.JsonConvert.DeserializeObject<List<MyObject>>(_stringBuilder.ToString());

It didn't like the ToString() method.

It also crashed with a simple line like

string _convertedString = _stringBuilder.ToString();

The full error is:

An exception of type 'System.OutOfMemoryException' occurred in mscorlib.dll but was not handled in user code

The machine is running 64bit windows with 16Gb of memory.

So, what are my options?

All I want is an IQueryable<MyObject> from a (very large) JSON string.

Check out the answer for this question, http://stackoverflow.com/questions/27315521/system-outofmemoryexception-with-json-net-with-listobject, deserializes one object at a time — dbugger, Dec 23 '15 at 13:34
An OutOfMemoryException can occur for various reasons, eg exhausting buffers, GDI objects etc. The exception's call stack will tell you where that occured, but the real culprit is inefficent code. *WHY* are you using a StringBuilder when you can read the data through a StreamReader? What you are doing isn't any better (or more efficient) than a `streamReader.ReadToEnd()`. — Panagiotis Kanavos, Dec 23 '15 at 13:40
@PanagiotisKanavos - those are just the various methods I've tried to get this to work just to show the processes. If there was a way to get the value in the StreamReader into the Deserializer without the memory error then I would love to know. — DeclanMcD, Dec 23 '15 at 13:43
If you create [a minimal, complete and verifiable example](http://stackoverflow.com/help/mcve) it will be easier for us to see the problem. — Orkun Bekar, Dec 23 '15 at 13:44
@dbugger - I didn't see that one. Never knew you could deserialize one item at a time. I will give it a go and see if that works. Thanks — DeclanMcD, Dec 23 '15 at 13:45

Panagiotis Kanavos · Accepted Answer · 2015-12-23T13:56:58.810

8

Your code essentially emulates what StreamReader.ReadToEnd does, taking at least 4 times the memory needed to read a large response (the memory of the string response itself, the StringBuilder's internal buffer, the size of all the intermediate temporary strings and the final string).

You can avoid this by deserializing from the stream directly with a JsonTextReader. Copying from the documentation sample:

using (var json= new JsonTextReader(streamReader))
{
    JsonSerializer serializer = new JsonSerializer();
    return (List<MyObject>)serializer.Deserialize(json, typeof(List<MyObject>));
}

O

edited Dec 23 '15 at 13:56

answered Dec 23 '15 at 13:50

Panagiotis Kanavos

120,703
13
188
236

perfect! worked a treat first time - just a minor change to your code - changed the word file to jsonTextReader – DeclanMcD Dec 23 '15 at 13:57

OutOfMemoryException when reading a string

1 Answers1

Linked