System.OutOfMemoryException when reading MemoryStream

Question

I am parsing local data to JSON, and then I parse JSON to XML. Parsing to JSON works fine, and I'm pretty sure that a week or two ago parsing to XML also worked fine - it wasn't in production, but I tested it and it worked. Now I get above-mentioned exception. Here's my code:

public static string ParseData(Data data)
{
    string xmlString = string.Empty;
    XmlData xmlData = new XmlData(data);

    using (MemoryStream memoryStream = new MemoryStream())
    using (StreamReader reader = new StreamReader(memoryStream))
    {
        xmlSerializer.WriteObject(memoryStream, xmlData);
        memoryStream.Position = 0;
        xmlString = reader.ReadToEnd(); //exception occurs here
    }
    return xmlString;
}

When exception occurs memoryStream.Position's value equals it's length and I still have like 200-300 mb in RAM. It's 64-bit app and 64-bit system. I checked similar questions, but in my case there is no big amounts of data(json is 1.5mb max).

UPDATED. Stack trace:

"at System.Text.StringBuilder.ToString()\r\n at System.IO.StreamReader.ReadToEnd()\r\n at Common.Util.LiveScoringXml.ParseData(Data data) in C:\Source\Repos\Latest\Common\Util\LiveScoringXml.cs:line 30"

If you have sql on your machine it might be eating all of you ram even if in your task manager it's seem that you have enough. In my case sometimes the system ran ok and sometimes it was throwing out of memory. If you have sql you should limit its memory usage — Simeon Vanov, Oct 09 '17 at 08:12
How about a `GC.Collect` right inside `ParseData` just to see whether this makes any difference. — Uwe Keim, Oct 09 '17 at 08:12
What do you do with the result of the function? Write it to disk? — Magnus, Oct 09 '17 at 08:13
@Jeric, I can send you json and class schema, you can deserialize json and you will get Data object, then you can run the function. — Jamil, Oct 09 '17 at 08:20

Ignacio Soler Garcia · Answer 1 · 2017-10-09T08:41:00.457

4

If you have a 64 bit application the most probably reason of your problem (apart from a bug in the .Net Framework) is that you are having the issue due to fragmentation of the Large Object Heap memory, any object bigger than 80k is stored there.

Check the following links:

Why Large Object Heap and why do we care?

https://blogs.msdn.microsoft.com/dotnet/2011/10/03/large-object-heap-improvements-in-net-4-5/

You are probably generating big chunks of data quite often and fragmenting the LOH until you cannot find a contiguous chunk big enough.

There are strategies to solve the issue like reusing objects instead of creating and GC'ing them.

edited Oct 09 '17 at 08:41

answered Oct 09 '17 at 08:28

Ignacio Soler Garcia

21,122
31
128
207

1

I never was brave enough to say "_…the only possible reason is…_" just about anything I came across in my whole life. – Uwe Keim Oct 09 '17 at 08:33
1

@UweKeim: ok, I will rephrase the answer so it is a bit less brave. Anyway it's something more cultural than anything, Spanish people talk different than Germans :) – Ignacio Soler Garcia Oct 09 '17 at 08:40
That sounds _way_ better. Thanks a lot . – Uwe Keim Oct 09 '17 at 08:41

System.OutOfMemoryException when reading MemoryStream

1 Answers1