c# serialization xmlwriter stringwriter out of memory for large objecthelp

Question

I am managing a large project and need to serialize and send an object in xml format. The object is ~130 mb.

(NOTE: I did not write this project, so making edits outside of this method, or drastically changing the architecture is not an option. It works great normally, but when the object is this large, it throws out of memory exception. I need to do it another way to handle large objects.)

The current code is this:

public static string Serialize(object obj)
{
    string returnValue = null;

    if (null != obj)
    {
        System.Runtime.Serialization.DataContractSerializer formatter = new System.Runtime.Serialization.DataContractSerializer(obj.GetType());

        XDocument document = new XDocument();

        System.IO.StringWriter writer = new System.IO.StringWriter();
        System.Xml.XmlTextWriter xmlWriter = new XmlTextWriter(writer);

        formatter.WriteObject(xmlWriter, obj);
        xmlWriter.Close();

        returnValue = writer.ToString();
    }

    return returnValue;
}

It is throwing an out of memory exception right at returnValue = writer.ToString().

I rewrote it to use "using" blocks which I prefer:

public static string Serialize(object obj)
    {
        string returnValue = null;
        if (null != obj)
        {
            System.Runtime.Serialization.DataContractSerializer formatter = new System.Runtime.Serialization.DataContractSerializer(obj.GetType());
            using (System.IO.StringWriter writer = new System.IO.StringWriter())
            {
                using (System.Xml.XmlTextWriter xmlWriter = new XmlTextWriter(writer))
                {
                    formatter.WriteObject(xmlWriter, obj);
                    returnValue = writer.ToString();
                }
            }
        }
        return returnValue;
    }

researching this, it appears the ToString method on StringWriter actually uses double the RAM. (I actually have plenty of RAM free, over 4 gb, so not really sure why I am getting an out of memory error).

Is the application build for 'x86', 'x64' or 'Any CPU'. If the build platform is x86 then your .net application is running as 32bit process although you have a 64bit environment and could access more memory if it would run as x64. — Ralf Bönning, Oct 16 '16 at 20:41
It's x86 and I can not change that. But still, it's a 130 meg object, so either way it doesn't go above the x86 memory limit. — jo phul, Oct 16 '16 at 20:45
On 32 bit the practical limit of how much *contiguous* memory you can allocate is much less than the 2GB theoretical limit. See http://stackoverflow.com/questions/2415434/the-limitation-on-the-size-of-net-array/2415472#2415472 for instance. What sort of changes are you able to make to this code? Do you have to return a string? Could it be something else? — dbc, Oct 16 '16 at 21:29
Don't use `StringWriter`. Don't return string. Use `FileStream`, write into it directly. — Alexander Petrov, Oct 17 '16 at 02:05
I often see code similar to this: huge amounts of data is serialized into a string, then this string is repeatedly passed from method to method. My hair stand on end from it! Strings in the .NET are copied by value. Holy shit! In the name of the garbage collector, CPU cycles and John Skeet's, leave the g-string to girls! — Alexander Petrov, Oct 17 '16 at 02:20
I agree. I took over a project with about half a million lines of code, written by idiots with no foresight. Works fine when the data packets are small, but now we are dealing with larger and larger data, and their methods break. Unfortunately, having return a file instead of a string means I have to now re-write tons of other methods that expect a string to be returned. I got it to serialize to a file, and have amended the few methods where that is an issue. Now it turns out, on the other end, deserializing is done to a string too! nightmare. — jo phul, Oct 17 '16 at 04:39

score 0 · Answer 1 · answered Oct 17 '16 at 23:10

0

Well, I found the best solution was to serialize to a file directly, then instead of passing a string along, I pass the file:

public static void Serialize(object obj, FileInfo destination)
{
    if (null != obj)
    {
        using (TextWriter writer = new StreamWriter(destination.FullName, false))
        {
            XmlTextWriter xmlWriter = null;
            try
            {
                xmlWriter = new XmlTextWriter(writer);
                DataContractSerializer formatter = new DataContractSerializer(obj.GetType());
                formatter.WriteObject(xmlWriter, obj);
            }
            finally
            {
                if (xmlWriter != null)
                {
                    xmlWriter.Flush();
                    xmlWriter.Close();
                }
            }
        }
    }
}

Of course, now I have another problem which I will post ... and that is deserializing the file!

answered Oct 17 '16 at 23:10

jo phul

639
1
9
29

You should dispose `xmlWriter` too. – Alexander Petrov Oct 18 '16 at 00:31
XmlWriter is not an IDisposable object. – jo phul Oct 18 '16 at 03:29
[1](https://msdn.microsoft.com/en-us/library/system.xml.xmlwriter(v=vs.110).aspx), [2](https://msdn.microsoft.com/en-us/library/system.xml.xmlwriter.dispose(v=vs.110).aspx). – Alexander Petrov Oct 18 '16 at 03:55
Weird. I am using .Net 4.5 and Visual Studio. If I type xmlWriter.Dispose() I get method not found. Also, if I put xmlWriter in using blocks - it crashes at that point. – jo phul Oct 19 '16 at 15:21

c# serialization xmlwriter stringwriter out of memory for large objecthelp

1 Answers1