Deserializing large json from WebService using Json.NET

Question

I'm receiving a large JSON string from a WebService and I'm looking for the best memory optimized way of deserializing it with C#.

JSON structure:

{
    "d": {
        "results": [
            {
                "metadata": {
                    "id": "",
                    "uri": "",
                    "type": ""
                },
                "ID": "",
                "Value1": "",
                "Value2": "",
                "Value3": ""
            },
            {
                "metadata": {
                    "id": "",
                    "uri": "",
                    "type": ""
                },
                "ID": "",
                "Value1": "",
                "Value2": "",
                "Value3": ""
            },
        ]
    }
}

I want to get all the object inside the "result" array but only one object after another and not like right now the complete list. I'm already using a StreamReader to avoid loading the complete json string into memory. Is there any option to read only one object, do some processing and then read the next one to avoid "OutOfMemoryExceptions"?

WebResponse response = r.GetResponse();  
using (Stream dataStream = response.GetResponseStream())
{
    var serializer = new JsonSerializer();

    using (var sr = new StreamReader(dataStream))
    using (var jsonTextReader = new JsonTextReader(sr))
    {
        return ((RootObject)serializer.Deserialize<RootObject>(jsonTextReader)).RootObject2.Results;
    }

Answer 1

What you can do is to adopt the basic approach of Issues parsing a 1GB json file using JSON.NET and Deserialize json array stream one item at a time, which is to stream through the JSON and deserialize and yield return each object; but in addition apply some stateful filtering expression to deserialize only StartObject tokens matching the path d.results[*].

To do this, first define the following interface and extension method:

public interface IJsonReaderFilter
{
    public bool ShouldDeserializeToken(JsonReader reader);
}

public static class JsonExtensions
{
    public static IEnumerable<T> DeserializeSelectedTokens<T>(Stream stream, IJsonReaderFilter filter, JsonSerializerSettings settings = null, bool leaveOpen = false)
    {
        using (var sr = new StreamReader(stream, leaveOpen : leaveOpen))
        using (var reader = new JsonTextReader(sr))
            foreach (var item in DeserializeSelectedTokens<T>(reader, filter, settings))
                yield return item;
    }

    public static IEnumerable<T> DeserializeSelectedTokens<T>(JsonReader reader, IJsonReaderFilter filter, JsonSerializerSettings settings = null)
    {
        var serializer = JsonSerializer.CreateDefault(settings);
        while (reader.Read())
            if (filter.ShouldDeserializeToken(reader))
                yield return serializer.Deserialize<T>(reader);
    }
}

Now, to filter only those items matching the path d.results[*], define the following filter:

class ResultsFilter : IJsonReaderFilter
{
    const string path = "d.results";
    const int pathDepth = 2;
    bool inArray = false;

    public bool ShouldDeserializeToken(JsonReader reader)
    {
        if (!inArray && reader.Depth == pathDepth && reader.TokenType == JsonToken.StartArray && string.Equals(reader.Path, "d.results", StringComparison.OrdinalIgnoreCase))
        {
            inArray = true;
            return false;
        }
        else if (inArray && reader.Depth == pathDepth + 1 && reader.TokenType == JsonToken.StartObject)
            return true;
        else if (inArray && reader.Depth == pathDepth && reader.TokenType == JsonToken.EndArray)
        {
            inArray = false;
            return false;
        }
        else
        {
            return false;
        }
    }
}

Next, create the following data model for each result:

public class Metadata
{
    public string id { get; set; }
    public string uri { get; set; }
    public string type { get; set; }
}

public class Result
{
    public Metadata metadata { get; set; }
    public string ID { get; set; }
    public string Value1 { get; set; }
    public string Value2 { get; set; }
    public string Value3 { get; set; }
}

And now you can deserialize your JSON stream incrementally as follows:

foreach (var result in JsonExtensions.DeserializeSelectedTokens<Result>(dataStream, new ResultsFilter()))
{
    // Process each result in some manner.
    result.Dump();
}

Demo fiddle here.