Out of memory exception while loading large json file from disk

Question

I have a 1.2 GB json file which when deserialized ought to give me a list with 15 mil objects.

The machine on which I'm trying to deserialize the same is a windows 2012 server(64 bit) with 16 core and 32 GB Ram.

The application has been built with target of x64.

Inspite of this when I try to read the json doc and convert it to list of objects I'm getting Out of memory exception. when I look at task manager I find that only 5GB memory has been used.

The codes i tried are as below..

a.

 string plays_json = File.ReadAllText("D:\\Hun\\enplays.json");

                plays = JsonConvert.DeserializeObject<List<playdata>>(plays_json);

b.

 string plays_json = "";
        using (var reader = new StreamReader("D:\\Hun\\enplays.json"))
        {
            plays_json = reader.ReadToEnd();
            plays = JsonConvert.DeserializeObject<List<playdata>>(plays_json);
        }

c.

 using (StreamReader sr = File.OpenText("D:\\Hun\\enplays.json"))
        {
            StringBuilder sb = new StringBuilder();
            sb.Append(sr.ReadToEnd());
            plays_json = sb.ToString();
            plays = JsonConvert.DeserializeObject<List<playdata>>(plays_json);
        }

All help is sincerely appreciated

Do you ever get as far as calling `DeserializeObject`? All your snippets first try to create a string which probably takes 2.4GB of memory, which I suspect is bigger than the CLR can manage for a single non-array object. (You *may* be able to do this with JsonReader though - avoid ever loading the whole string at once.) — Jon Skeet, Oct 10 '16 at 09:41
Otherwise, perhaps you need to use [``](https://msdn.microsoft.com/en-us/library/hh285054(v=vs.110).aspx) (if an array is exceeding 2GB in size) — Matthew Watson, Oct 10 '16 at 09:42
You really should have a read of this http://www.drdobbs.com/windows/parsing-big-records-with-jsonnet/240165316 — Paul Zahra, Oct 10 '16 at 09:59
See posting : http://stackoverflow.com/questions/21151125/deserializing-large-json-objects-from-web-service-out-of-memory — jdweng, Oct 10 '16 at 10:00

Brian Rogers · Accepted Answer · 2017-08-22T14:43:12.583

The problem is that you are reading your entire huge file into memory and then trying to deserialize it all at once into a huge list. You should be using a StreamReader to process your file incrementally. Example (b) in your question doesn't cut it, even though you are using a StreamReader there, because you are still reading the entire file via ReadToEnd(). You should be doing something like this instead:

using (StreamReader sr = new StreamReader("D:\\Hun\\enplays.json"))
using (JsonTextReader reader = new JsonTextReader(sr))
{
    var serializer = new JsonSerializer();

    while (reader.Read())
    {
        if (reader.TokenType == JsonToken.StartObject)
        {
            // Deserialize each object from the stream individually and process it
            var playdata = serializer.Deserialize<playdata>(reader);

            ProcessPlayData(playdata);
        }
    }
}

The ProcessPlayData method should process a single playdata object and then ideally write the result to a file or a database rather than an in-memory list (otherwise you may find yourself back in the same situation again). If you must store the results of processing each item into an in-memory list, then you might want to consider using a linked list or a similar structure that does not try to allocate memory in one contiguous block and does not need to reallocate and copy when it needs to expand.

But my requirement is to do complex calculations and linq queries on the list of all playdata objects.. though thanks for the JsonTextReader, I will try that out.. — Arnab, Oct 11 '16 at 02:50
Excellent! The `while (reader.Read()) {` with the included single object deserialization brought my memory consumption from 13GB down to 250MB. Thumbs up! — z00l, Jan 05 '18 at 13:53
this is some answer .... awesome. i am always forgetting that there are people out there that are killing this c#. i got 2.5gb file and this incremental opening works as expected. awesome job. — pregmatch, May 04 '22 at 10:57

score 2 · Answer 2 · edited May 23 '17 at 12:33

In my opinion, your out of memory exception can be due to one of the below reasons.

The size of your object plays is over 2GB, and by default the maximum size of a CLR object in .NET is 2GB(even on x64) See here

Now, your object doesn't have to be 2GB. Fragmentation in Large Object Heap (LOH) can cause an object smaller than 2GB to throw an out of memory exception as well. (Any object over 80kb or so will reside in Large object heap)

Another case is when OS can't allocate a contiguous block of virtual memory for your large object, but I don't think this would be the case since you've mentioned you have 32GB RAM.

I wouldn't just go and enable gcAllowVeryLargeObjects unless there aren't any other options. I have seen memory consumption of one of my large data handling Apis go up from 3GB to 8GB after turning on that setting. (Even though most of it were only reserved) I think this is because you are allowing your app to ask the OS for as much memory as it needs to hold a large object. This can be particularly problematic if you are hosting other apps on the same server. It's good to have an upper limit on how much memory a managed object can take.

Another thing to note is that by default GC doesn't compact the LOH. So, this means the size of the working set will remain large unless a full garbage collection takes place. (You can call GC to compact LOH from .NET 4.5.1 onwards) See here

I would strongly recommend using a memory profiler like dotMemory to first understand what's going on under the hood before making any decisions.

If you are targeting x64 and if this is a web application, then make sure IIS is set to use 64-bit version as well. See here for local IIS express and IIS on Server

If I were you, I would try to break this task into smaller batches.

What is the purpose of loading this entire file at one go? Are you trying to do some IO operation with the loaded data or any CPU intensive tasks?

Here is a useful link on GC fundamentals

Out of memory exception while loading large json file from disk

2 Answers2

Linked

Related