Out of memory exception reading "Large" file

Question

I'm trying to serialize an object into a string The first problem I encountered was that the XMLSerializer.Serialize method threw an Out of memory exception, I've trying all kind of solutions and none worked so I serialized it into a file. The file is about 300mb's (32 bit process, 8gb ram) and trying to read it with StreamReader.ReadToEnd also results in Out of memory exception.

The XML format and loading it on a string are not an option but a must. The question is:

• Any reason that a 300mb file will throw that kind of exception? 300mb is not really a large file.

Serialization code that fails on .Serialize

using (MemoryStream ms = new MemoryStream())
{
    var type = obj.GetType();
    if (!serializers.ContainsKey(type))
        serializers.Add(type,new XmlSerializer(type));

   // new XmlSerializer(obj.GetType()).Serialize(ms, obj);
    serializers[type].Serialize(ms, obj);
    ms.Position = 0;

    using (StreamReader sr = new StreamReader(ms))
    {
        return sr.ReadToEnd();
    }                  
}

Serialization and read from file that fails on ReadToEnd var type = obj.GetType(); if (!serializers.ContainsKey(type)) serializers.Add(type,new XmlSerializer(type));

FileStream fs = new FileStream(@"c:/temp.xml", FileMode.Create);
TextWriter writer = new StreamWriter(fs, new UTF8Encoding());
serializers[type].Serialize(writer, obj);
writer.Close();
fs.Close();
using (StreamReader sr = new StreamReader(@"c:/temp.xml"))
{
   return sr.ReadToEnd();
}

The object is large because its an elaborate system entire configuration object...

UPDATE: Reading the file in chucks (8*1024 chars) will load the file into a StringBuilder but the builders fails on ToString().... starting to think there is no way which is really strange.

Answer 1

Yeah, if you're using 32-bit, trying to load 300MB in one chunk is going to be awkward, especially when using approaches that don't know the final size (number of characters, not bytes) in advance, thus have to keep doubling an internal buffer. And that is just when processing the string! It then needs to rip that into a DOM, which can often take several times as much space as the underlying data. And finally, you need to deserialize it into the actual objects, usually taking about the same again.

So - indeed, trying to do this in 32-bit will be tough.

The first thing to try is: don't use ReadToEnd - just use XmlReader.Create with either the file path or the FileStream, and let XmlReader worry about how to load the data. Don't load the contents for it.

After that... the next thing to do is: don't limit it to 32-bit.

Well, you could try enabling the 3GB switch, but... moving to 64-bit would be preferable.

Aside: xml is not a good choice for large volumes of data.

Answer 2

Exploring the source code for StreamReader.ReadToEnd reveals that it internally makes use of the StringBuilder.Append method:

public override String ReadToEnd()
{
    if (stream == null)
        __Error.ReaderClosed();

#if FEATURE_ASYNC_IO
    CheckAsyncTaskInProgress();
#endif

    // Call ReadBuffer, then pull data out of charBuffer.
    StringBuilder sb = new StringBuilder(charLen - charPos);
    do {
        sb.Append(charBuffer, charPos, charLen - charPos);
        charPos = charLen;  // Note we consumed these characters
        ReadBuffer();
    } while (charLen > 0);
    return sb.ToString();
}

which most probably throws this exception that leads to the this question/answer: interesting OutOfMemoryException with StringBuilder al