Deserializing large JSON Objects from Web Service (Out of Memory)

Question

I have a program that deserializes large objects from a web service. After a webservice call and a 200, the code looks like this.

JsonConvert.DeserializeObject<List<T>>(resp.Content.ReadAsStringAsync().Result).ToList()

Sometimes while running this process I will get an aggregate exception which shows an inner exception as out of memory. I can't determine if it is the process of reading in the string of JSON data (which is probably awfully large) or the Deserializing that is causing this issue. What I would like to do is break out the string and pull each JSON object back individually from the response and then deserialize it. I am just having trouble finding a way to only bring out one JSON object at a time from the response. Any suggestions are greatly appreciated!

Serialization is failing because, and this an example, you try to assign a string to a number. It happen with booleans or any other type — Dalorzo, Jan 16 '14 at 00:31
The serialization only fails sometimes and it throws an out of memory exception. Since it it intermittent with the same JSON data I wouldn't believe this is anything to do with type. — user3170736, Jan 16 '14 at 00:33
Is this a 32 bit process? Have you observed memory usage in Task Manager? — Cam Bruce, Jan 16 '14 at 00:37
http://stackoverflow.com/questions/8630736/getting-an-outofmemoryexception-while-serialising-to-json — Dalorzo, Jan 16 '14 at 00:38
Any way to change it to a 64bit process? That should help, assuming you were running out of memory because of the 32bit limit. — Cam Bruce, Jan 16 '14 at 00:41
No, thats not really feasible given the device the code will run on. — user3170736, Jan 16 '14 at 00:42
Would be great if you could share your answer. That's what Stackoverflow is about. — DanielG, Nov 02 '15 at 12:10

score 4 · Accepted Answer · answered Jan 16 '14 at 00:46

4

HttpClient client = new HttpClient();

using (Stream s = client.GetStreamAsync("http://www.test.com/large.json").Result)
using (StreamReader sr = new StreamReader(s))
using (JsonReader reader = new JsonTextReader(sr))
{
    JsonSerializer serializer = new JsonSerializer();

    // read the json from a stream
    // json size doesn't matter because only a small piece is read at a time from the HTTP request
    Person p = serializer.Deserialize<Person>(reader);
}

answered Jan 16 '14 at 00:46

user3170736

511
5
24

1

Why use `GetStreamAsync(...).Result`? Waiting for an uncompleted task result blocks, so you might as well use the synchronous version. – spender Jan 16 '14 at 01:47
2

You should probably reference where you got the code from as well. – nick_w Jan 16 '14 at 02:50
Code came from http://james.newtonking.com/json/help/index.html?topic=html/Performance.htm. As for the async versus sync versions the site says to use async. – user3170736 Jan 16 '14 at 14:41
The user example returns a List. If the list is too large, this won't help. See this answer: http://stackoverflow.com/a/24115672/475876 – Akira Yamamoto Jun 21 '16 at 15:17

score 0 · Answer 2 · answered May 17 '22 at 07:52

https://learn.microsoft.com/en-us/xamarin/xamarin-forms/data-cloud/web-services/rest contains a warning:

Using the ReadAsStringAsync method to retrieve a large response can have a negative performance impact. In such circumstances the response should be directly deserialized to avoid having to fully buffer it.

Deserializing large JSON Objects from Web Service (Out of Memory)

2 Answers2

Linked