Json.net: Preventing OutOfMemoryErrors when handling large objects

Question

My question is basically the same as this one, but unfortunately the answer doesn't work for me.

I'm trying to handle some edge cases, one of which is the case where I receive an unusually large (e.g., 150 million character) json string from my server. I'm using the memory optimization technique described in the json.net docs, but I still get an OutOfMemoryError in the deserialization step.

What is the correct way to handle large json objects with json.net? I'd settle for setting a max length flag, but I'm not sure how to do that.

Here's my code:

    Dim serverResponseStream As Stream = 'GZipWrapperStream response from server

    Using sr As StreamReader = New StreamReader(serverResponseStream), _
        reader As JsonReader = New JsonTextReader(sr)
        Dim serializer As JsonSerializer = New JsonSerializer()

        'Out of memory exception here in Deserialize
        Dim response = serializer.Deserialize(reader) 
    End Using

Edit: as per this answer, I have also tried:

    Dim a = New JArray()
    Using sr As StreamReader = New StreamReader(serverResponseStream), _
            reader As JsonReader = New JsonTextReader(sr)
        While reader.Read()
            If reader.TokenType = JsonToken.StartObject Then

                'Out of memory exception here
                Dim j = JObject.Load(reader)

                a.Add(j)                    
            End If
        End While
    End Using

AFAIK you'll have to use JsonReader to manually "read and unwrap" the top-level objects such that you only Deserialize (and use) smaller parts of the object-graph at once - the use of JsonTextReader around a string is to allow the stream to be consumed directly, which can avoid an intermediate String but it will *not* lower the overall usage of Deserialize. (Alternatively, bump up the VM and cross yer fingers xD) — user2864740, May 30 '14 at 04:16
See http://www.drdobbs.com/windows/parsing-big-records-with-jsonnet/240165316 — user2864740, May 30 '14 at 04:19
And you may be able to find some links/information off of http://stackoverflow.com/questions/14990590/is-there-a-pull-style-json-streamreader-parser-for-net — user2864740, May 30 '14 at 04:21
Thanks for the links. I should have included that article in the question--I read it but was hoping not to have to resort to that technique. — ForeverWintr, May 30 '14 at 04:54
possible duplicate of [Deserialize json array stream one item at a time](http://stackoverflow.com/questions/20374083/deserialize-json-array-stream-one-item-at-a-time) — Brian Rogers, Jun 02 '14 at 01:26
@BrianRogers, how is your answer in the question you linked different than the one that I linked to? I still get an out of memory error on the line `JObject obj = JObject.Load(reader)` — ForeverWintr, Jun 02 '14 at 20:10
@ForeverWintr In the question you linked to, the entire JSON is deserialized in one go into the `Person` object. In the one I linked to, a while loop is used to only deserialize one object at a time from an array. You never said what your JSON looks like-- I was assuming it would be a list of large objects. If you have just a single large object then the second answer reduces to be the same as the first answer. In that case, you'd need to modify the second answer to work on smaller parts of the JSON, e.g. one property at a time. It might be helpful if you shared a sample of your JSON. — Brian Rogers, Jun 02 '14 at 20:22
@ForeverWintr I also notice in your edited answer, that you are adding all of the deserialized objects to a JArray, which keeps them all in memory. If you're trying to save memory, you don't want to do that. Process one at a time if you can, then let that memory be reclaimed. — Brian Rogers, Jun 02 '14 at 20:29
@BrianRogers, Hmm, you're right. I added the JArray to test your suggestion, assuming that the memory error would occur on `a.Add(j)` if the stored array was the cause of the issue. It looks like that assumption was wrong, as if I remove the array I don't run out of memory. I guess that isn't too surprising in hindsight. — ForeverWintr, Jun 02 '14 at 21:06

score -1 · Answer 1 · answered May 30 '14 at 08:46

-1

This might work (depending on the size of json you're deserilizing) :

serializer.MaxJsonLength = Int32.MaxValue;

Edit : my bad this is if you're using JavaScriptSerializer()

answered May 30 '14 at 08:46

dekajoo

2,024
1
25
36

2

Yeah, If there's an equivalent property in a JsonSerializer I'd like to know what it is. Unfortunately I haven't been able to find one. – ForeverWintr May 30 '14 at 18:42

Json.net: Preventing OutOfMemoryErrors when handling large objects

1 Answers1