Strings appear to be sticking around too long

Question

In short, I've got an application that converts a flat data file into an XML file. It does this by populating objects and then serializing them to XML.

The problem I'm running into is that the Garbage Collector does not seem to be taking care of the serialized strings. 3500 record files are running up to OutOfMemoryExceptions before they finish. Something is fishy, indeed.

When I take the serialization out of the mix and simply pass an empty string, the memory consumption remains as expected, so I've ruled out the possibility that my intermediate objects (between flat file and xml) are the problem here. They seem to be collected as expected.

Can anyone help? How do I make sure these strings are disposed of properly?

Update: Some sample code

// myObj.Serialize invokes an XmlSerializer instance to handle its work    
string serialized = myObj.Serialize();
myXmlWriter.WriteRaw(serialized);

This is basically where the problem is ocurring--if I take the string serialized out of play, the memory problems go away, too, even though I'm still transforming the flat file into objects, one at a time.

Update 2: Serialize method

public virtual string Serialize()
{
      System.IO.StreamReader streamReader = null;
      System.IO.MemoryStream memoryStream = null;

      using (memoryStream = new MemoryStream())
      {
          memoryStream = new System.IO.MemoryStream();
          Serializer.Serialize(memoryStream, this);

          memoryStream.Seek(0, System.IO.SeekOrigin.Begin);
          using (streamReader = new System.IO.StreamReader(memoryStream))
          {
              return streamReader.ReadToEnd();
          }
      }
}

try using StringBuilder and reusing it, also with that many records perhaps saving them on the file system/DB would work too — Kris Ivanov, Feb 03 '11 at 14:45
I agree, but I work for a large corporation, and can't post proprietary code. — Brian Warshaw, Feb 03 '11 at 14:46
K Ivanov--I'll try a StringBuilder. As for number of records, I'm already writing each record to the fs after it is serialized. — Brian Warshaw, Feb 03 '11 at 14:47
Are you allocating big things? I'm thinking LOH fragmentation may be an issue - http://stackoverflow.com/questions/686950/large-object-heap-fragmentation — dsolimano, Feb 03 '11 at 14:54
No, dsolimano, my largest objects are being disposed properly. As I mention above, if I take the serialization out of the equation (which creates big strings), then memory is fine, even with my largest objects still being created. — Brian Warshaw, Feb 03 '11 at 14:59
Thanks for code. Could please also explain what does this line do? Serializer.Serialize(memoryStream, this); — Andrey Taptunov, Feb 03 '11 at 15:10
Serializer is an instance of System.Xml.Serialization.XmlSerializer. That line is simply invoking the Serialize method of the instance. — Brian Warshaw, Feb 03 '11 at 15:20

score 2 · Accepted Answer · answered Feb 03 '11 at 14:50

You need to make sure they aren't referenced anywhere. Before an OutOfMemoryException is thrown, the GC is run. If it isn't recovering that memory, that means something is still holding on to it. Like others said, if you post some code, we might be able to help. Otherwise you can use a profiler or WinDbg/SOS to help figure out what is holding onto your strings.

score 1 · Answer 2 · answered Feb 03 '11 at 15:48

Very curious indeed. I added the following dandy after each serialized record writes to the XmlWriter:

if (GC.GetTotalMemory(false) > 104857600)
{
     GC.WaitForPendingFinalizers();
}

and wouldn't you know it, it's keeping it in check and it's processing without incident, never getting too far above the threshold I set. I feel like there should be a better way, but it almost seems like the code was executing too fast for the garbage collector to reclaim the strings in time.

score 0 · Answer 3 · answered Feb 03 '11 at 14:46

Do you have an example of your code - how you're creating these strings? Are you breaking out into unmanaged code anywhere (which means you would be required to clean-up after yourself).

Another thought is how you are converting flat data file into XML. XML can be somewhat heavy depending on how you are building the file. If you are trying to hold the entire object in memory, it is very likely (easy to do, in fact) that you are running out of memory.

score 0 · Answer 4 · answered Feb 03 '11 at 20:43

It sure looks like your method could be cleaned up to be just:

public virtual string Serialize()
{
    StringBuilder sb = new StringBuilder();
    using (StringWriter writer = new StringWriter(sb))
    {
        this.serializer.Serialize(writer, this);
    }

    return sb.ToString();
}

You are creating an extra MemoryStream for no reason.

But if you are writing the string to a file, then why don't you just send a FileStream to the Serialize() method?

Strings appear to be sticking around too long

4 Answers4