JsonConvert Deserialize Object out of memory exception

Question

I am trying to deserialize an object received through http web response and it's working fine when the byte[] within the object is of less size. But when the size increases JsonConvert Deserializing is throwing out of memory exception. Input is the http web response not a file. This is deserializing a single object that is too large.

var response = (HttpWebResponse)request.GetResponse();
if (response.StatusCode == HttpStatusCode.OK)
{
    using (var streamReader = new StreamReader(response.GetResponseStream()))
    {
        return JsonConvert.DeserializeObject<SomeObjectReply<TResponse>>(streamReader.ReadToEnd());
    }
}

I have tried the following but it is still the same issue

if (response.StatusCode == HttpStatusCode.OK)
{
    using (var streamReader = new StreamReader(response.GetResponseStream()))
    using (var jsonReader = new JsonTextReader(streamReader))
    {
        JsonSerializer serializer = new JsonSerializer();
        return (SomeObjectReply<TResponse>)serializer.Deserialize(jsonReader, typeof(SomeObjectReply<TResponse>));
    }
}   

The JSON looks like:

{
    "Id":"81a130d2-502f-4cf1-a376-63edeb000e9f",
    // The following is MUCH larger in cases where the exception is thrown.
    "Document":"AAECAwQFBgcICQoLDA0ODxAREhMUFRYXGBkaGxwdHh8gISIjJCUmJygpKissLS4vMDEyMzQ1Njc4OTo7PD0+P0BBQkNERUZHSElKS0xNTk9QUVJTVFVWV1hZWltcXV5fYGFiY2RlZmdoaWprbG1ub3BxcnN0dXZ3eHl6e3x9fn+AgYKDhIWGh4iJiouMjY6PkJGSk5SVlpeYmZqbnJ2en6ChoqOkpaanqKmqq6ytrq+wsbKztLW2t7i5uru8vb6/wMHCw8TFxsfIycrLzM3Oz9DR0tPU1dbX2Nna29zd3t/g4eLj5OXm5+jp6uvs7e7v8PHy8/T19vf4+fr7/P3+/wABAgMEBQYHCAkKCwwNDg8QERITFBUWFxgZGhscHR4fICEiIyQlJicoKSorLC0uLzAxMjM0NTY3ODk6Ozw9Pj9AQUJDREVGR0hJSktMTU5PUFFSU1RVVldYWVpbXF1eX2BhYmNkZWZnaGlqa2xtbm9wcXJzdHV2d3h5ent8fX5/gIGCg4SFhoeIiYqLjI2Oj5CRkpOUlZaXmJmam5ydnp+goaKjpKWmp6ipqqusra6vsLGys7S1tre4ubq7vL2+v8DBwsPExcbHyMnKy8zNzs/Q0dLT1NXW19jZ2tvc3d7f4OHi4+Tl5ufo6err7O3u7/Dx8vP09fb3+Pn6+/z9/v8="
}

After deserialization, the object will be

public class Doc
{
    public Guid Id { get; set; }
    public byte[] Document { get; set; }
}

I.e. I am deserializing a single object with large byte array. And when the Document byte array property becomes too large, I get the exception:

Message: Exception of type 'System.OutOfMemoryException' was thrown.
Stack Trace:
   at Newtonsoft.Json.Utilities.BufferUtils.RentBuffer(IArrayPool`1 bufferPool, Int32 minSize)
   at Newtonsoft.Json.JsonTextReader.PrepareBufferForReadData(Boolean append, Int32 charsRequired)
   at Newtonsoft.Json.JsonTextReader.ReadData(Boolean append, Int32 charsRequired)
   at Newtonsoft.Json.JsonTextReader.ReadStringIntoBuffer(Char quote)
   at Newtonsoft.Json.JsonTextReader.ReadAsBytes()
   at Newtonsoft.Json.JsonReader.ReadForType(JsonContract contract, Boolean hasConverter)
   at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.PopulateObject(Object newObject, JsonReader reader, JsonObjectContract contract, JsonProperty member, String id)
   at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.CreateObject(JsonReader reader, Type objectType, JsonContract contract, JsonProperty member, JsonContainerContract containerContract, JsonProperty containerMember, Object existingValue)
   at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.CreateValueInternal(JsonReader reader, Type objectType, JsonContract contract, JsonProperty member, JsonContainerContract containerContract, JsonProperty containerMember, Object existingValue)
   at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.SetPropertyValue(JsonProperty property, JsonConverter propertyConverter, JsonContainerContract containerContract, JsonProperty containerProperty, JsonReader reader, Object target)
   at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.PopulateObject(Object newObject, JsonReader reader, JsonObjectContract contract, JsonProperty member, String id)
   at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.CreateObject(JsonReader reader, Type objectType, JsonContract contract, JsonProperty member, JsonContainerContract containerContract, JsonProperty containerMember, Object existingValue)
   at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.CreateValueInternal(JsonReader reader, Type objectType, JsonContract contract, JsonProperty member, JsonContainerContract containerContract, JsonProperty containerMember, Object existingValue)
   at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.Deserialize(JsonReader reader, Type objectType, Boolean checkAdditionalContent)
   at Newtonsoft.Json.JsonSerializer.DeserializeInternal(JsonReader reader, Type objectType)
   at Newtonsoft.Json.JsonSerializer.Deserialize[T](JsonReader reader)

Answer 1

Your traceback indicates that Json.NET is running out of memory trying to materialize a single property, namely a byte [] property corresponding to a single Base64 string. If your JSON contains a string value that cannot be materialized as a .Net type without running out of memory then you cannot use Json.NET to parse your JSON because JsonTextReader always fully materializes each property - even when those properties would otherwise be skipped.

As an alternative, you might consider using the reader returned by JsonReaderWriterFactory.CreateJsonReader() to manually parse your JSON. This factory returns a XmlDictionaryReader that transcodes from JSON to XML on the fly, and thus supports incremental reading of Base64 properties via XmlReader.ReadContentAsBase64(Byte[], Int32, Int32). Using this approach you could manually copy the Document property into a Stream that allows for large amounts of data such as a FileStream or a RecyclableMemoryStream.

First, define the following data model:

public class StreamedDocument : IDisposable
{
    public Guid Id { get; set; }
    public Stream Document { get; init; }

    public void Dispose() => Document?.Dispose();
}

Then define the following factory for creating the data model above:

public static class DocumentFactory
{
    const int BufferSize = 8192;
    private static readonly Microsoft.IO.RecyclableMemoryStreamManager manager = new ();

    public static Stream CreateTemporaryStream() => 
        // Create some temporary stream to hold the document.  
        // Could be a FileStream created with FileOptions.DeleteOnClose or a Microsoft.IO.RecyclableMemoryStream
        //File.Create(Path.GetTempFileName(), BufferSize, FileOptions.DeleteOnClose);
        manager.GetStream();
    
    public static StreamedDocument CreateStreamedDocument(Stream inputStream) =>
        DocumentFactory.PopulateStreamedDocument(new StreamedDocument{ Document = CreateTemporaryStream() }, inputStream);
    
    public static StreamedDocument PopulateStreamedDocument(StreamedDocument doc, Stream inputStream)
    {
        if (doc == null)
            throw new ArgumentNullException();
        if (doc.Document == null)
            throw new ArgumentException("null doc.Document");
        using (var reader = JsonReaderWriterFactory.CreateJsonReader(inputStream, XmlDictionaryReaderQuotas.Max))
        {
            while (!reader.EOF)
            {
                if (reader.NodeType == XmlNodeType.Element && reader.LocalName == nameof(doc.Id))
                {
                    doc.Id = reader.ReadElementContentAsGuid();
                    // reader should now be positioned PAST the EndElement.
                    Debug.Assert(reader.NodeType != XmlNodeType.EndElement, $"reader.NodeType {reader.NodeType} != XmlNodeType.EndElement");
                }
                else if (reader.NodeType == XmlNodeType.Element && reader.LocalName == nameof(doc.Document))
                {
                    byte[] buffer = new byte[BufferSize];
                    int readBytes = 0;
                    while ((readBytes = reader.ReadElementContentAsBase64(buffer, 0, buffer.Length)) > 0)
                        doc.Document.Write(buffer, 0, readBytes);
                    // reader should now be positioned ON the EndElement
                    Debug.Assert(reader.NodeType == XmlNodeType.EndElement, "reader.NodeType == XmlNodeType.EndElement");
                    if (doc.Document.CanSeek)
                        doc.Document.Position = 0;
                }
                else
                {
                    reader.Read();
                }
            }
        }
        return doc;
    }
}

And now you should be able to do:

var response = (HttpWebResponse)request.GetResponse();
if (response.StatusCode == HttpStatusCode.OK)
{
    return DocumentFactory.CreateStreamedDocument(response.GetResponseStream());
}               

Notes:

• You must eventually dispose of the StreamedDocument to delete the temporary Stream Document.

• For a discussion of how the reader returned by JsonReaderWriterFactory transcodes from JSON to XML, see Efficiently replacing properties of a large JSON using System.Text.Json and Mapping Between JSON and XML.

• JsonReaderWriterFactory is used by DataContractJsonSerializer to serialize data contract classes from and to JSON. While not recommended for new development (Microsoft recommends to use system.text.json instead), this serializer and its reader have been fully ported to .NET Core / .NET 5 and may be used when circumstances justify, as they may do here.

• You might also check to see whether JsonSerializer from System.Text.Json is able to deserialize your JSON without running out of memory:

  var doc = await JsonSerializer.DeserializeAsync<Doc>(responseStream);
  

  Do note that System.Text.Json only supports async deserialization from streams.

  Demo fiddle here.

• RecyclableMemoryStream comes from the Microsoft.IO.RecyclableMemoryStream nuget package. It is a pooled MemoryStream allocator to decrease GC load and improve performance on highly scalable systems. Among other advantages it is a drop-in replacement for MemoryStream that does not try to allocate the stream contents as a single huge array.

Demo fiddle here.

Answer 2

Serializing/deserializing in memory can really create these types of issues when dealing with huge objects. As a workaround for myself I am always using file system as an intermediate storage.

Save your json to the temporary file and then deserialize it using these generic methods:

public static class Helper
{
    public static T CreateObjectFromJsonFile<T>(string filePath) where T : class
    {
        T obj;
        using (var file = File.OpenText(filePath))
        {
            var serializer = new JsonSerializer { NullValueHandling = NullValueHandling.Ignore };
            obj = (T)serializer.Deserialize(file, typeof(T));
        }

        return obj;
    }

    public static void CreateJsonFileFromObject(object input, string fileName)
    {
        using (var file = File.CreateText(fileName))
        {
            var serializer = new JsonSerializer { NullValueHandling = NullValueHandling.Ignore };
            serializer.Serialize(file, input);
        }
    }
}

Here is a full test example:

//create object
var list = new List<TvShow>
{
    new TvShow { Id = 0, Rating = 10, Title = "t1"},
    new TvShow { Id = 1, Rating = 20, Title = "t2"},
    new TvShow { Id = 2, Rating = 30, Title = "t3"}
};

//serialize to JSON and save to file system
const string filePath = "C://temp//test.txt";
Helper.CreateJsonFileFromObject(list, filePath);

//deserialize back to object from file system
var result = Helper.CreateObjectFromJsonFile<List<TvShow>>(filePath);

With your particular code:

const string filePath = "C://temp//test.txt";

//save JSON to file
var response = (HttpWebResponse)request.GetResponse();
if (response.StatusCode == HttpStatusCode.OK)
{
    using (var fileStream = new FileStream(filePath, FileMode.Create, FileAccess.Write))
    {
        response.Content.ReadAsStream().CopyTo(fileStream);
    }
}

//deserialize object from file
var result = Helper.CreateObjectFromJsonFile<SomeObjectReply<TResponse>>(filePath);