Serializing to MemoryStream causes an OutOfmemoryException, but serializing to a FileStream does not. Can anyone tell me why?

Question

I'm using Newtonsoft Json.Net to serialize objects as json. I continually run into an OutOfMemoryException when I try to serialize to a MemoryStream, but not when I serialize to a FileStream. Could someone explain why this might be happening? These are the two methods that I am using to serialize.

Throws an OutOfMemoryException

        private static MemoryStream _serializeJson<T>(T obj)
    {
        try
        {
            var stream = new MemoryStream();
            var streamWriter = new StreamWriter(stream);
            var jsonWriter = new JsonTextWriter(streamWriter);
            var serializer = new JsonSerializer();
            serializer.ContractResolver = new CamelCasePropertyNamesContractResolver();
            serializer.Formatting = Formatting.Indented;
            serializer.Serialize(jsonWriter, obj);
            streamWriter.Flush();
            stream.Position = 0;
            return stream;
        }
        catch (Exception e)
        {
            //Logger.WriteError(e.ToString());
            Console.WriteLine(e.ToString());
            return null;
        }
    }

Doesn't throw an OutOfMemoryException

    private static void _serializeJsonToFile<T>(T obj, string path)
    {
        try
        {
            using (FileStream fs = File.Open(path, FileMode.Create, FileAccess.ReadWrite))
            using (StreamWriter sw = new StreamWriter(fs))
            using (JsonWriter jw = new JsonTextWriter(sw))
            {
                jw.Formatting = Formatting.Indented;
                JsonSerializer serializer = new JsonSerializer();
                serializer.ContractResolver = new CamelCasePropertyNamesContractResolver();
                serializer.Serialize(jw, obj);
            }
        }
        catch (Exception e)
        {
            Console.WriteLine(e.ToString());
        }
    }

P.S. some might ask why I would want to return a stream instead of simply serializing to the file stream. This is because I want to keep serialization in one class and file handling in another, so I'm passing the memory stream to a WriteFile method in another class later.

Try to Dispose `MemoryStream`, `StreamWriter` and `JsonTextWriter`. Of course `MemoryStream` in consumer class/method. — Hamlet Hakobyan, Aug 03 '15 at 20:44
Refer to the excellent answer in this post:http://stackoverflow.com/questions/15595061/outofmemoryexception-while-populating-memorystream-256mb-allocation-on-16gb-sys — Jose Gomez, Aug 03 '15 at 20:49

score 2 · Accepted Answer · answered Aug 03 '15 at 20:55

You are getting OutOfMemoryExceptions because the memory stream is very aggressive about it's growth. Every time it needs to re-size it doubles it's internal buffer.

//The code from MemoryStream http://referencesource.microsoft.com/mscorlib/system/io/memorystream.cs.html#1416df83d2368912
private bool EnsureCapacity(int value) {
    // Check for overflow
    if (value < 0)
        throw new IOException(Environment.GetResourceString("IO.IO_StreamTooLong"));
    if (value > _capacity) {
        int newCapacity = value;
        if (newCapacity < 256)
            newCapacity = 256;
        // We are ok with this overflowing since the next statement will deal
        // with the cases where _capacity*2 overflows.
        if (newCapacity < _capacity * 2)
            newCapacity = _capacity * 2;
        // We want to expand the array up to Array.MaxArrayLengthOneDimensional
        // And we want to give the user the value that they asked for
        if ((uint)(_capacity * 2) > Array.MaxByteArrayLength)
            newCapacity = value > Array.MaxByteArrayLength ? value : Array.MaxByteArrayLength;

        Capacity = newCapacity;
        return true;
    }
    return false;
}

With a 17.8 MB file that is a worst case scenario of a 35.6 MB byte array being used. The old byte arrays that where discarded during the resizing process can also cause Memory Fragmentation depending on how long they live, this can easily get your program to throw a OOM error before you get to the 32 bit memory limit.

Writing directly to a FileStream does not require any large buffers to be created in memory so it uses much less space.

There is a way to separate the logic of the saving from the serializing, just pass in the stream to the function instead of creating it in the function itself.

private static void _serializeJson<T>(T obj, Stream stream)
{
    try
    {
        using(var streamWriter = new StreamWriter(stream, Encoding.UTF8, 1024, true))
        using(var jsonWriter = new JsonTextWriter(streamWriter))
        {
            var serializer = new JsonSerializer();
            serializer.ContractResolver = new CamelCasePropertyNamesContractResolver();
            serializer.Formatting = Formatting.Indented;
            serializer.Serialize(jsonWriter, obj);
         }
    }
    catch (Exception e)
    {
        //Logger.WriteError(e.ToString());
        Console.WriteLine(e.ToString());
    }
}

I also dispose of the StreamWriter that is created, the constructor I used has a leaveOpen flag which causes the underlying stream to not be closed when you dispose of the StreamWriter.

Serializing to MemoryStream causes an OutOfmemoryException, but serializing to a FileStream does not. Can anyone tell me why?

1 Answers1