Memory problems in .NET

Question

I have a C# service which listens to a queue for XML messages, receives them, processing them using XSLTs and writing them in the database. It does process about 60K messages a day of about 1Mb each. The memory when is idle is going down to 100MB which is really good. However recently I have started processing messages of 12 MB in size. It does blow the memory up and even when is idle it has memory of about 500MB. Any suggestions why this might be a problem? I don't think there is a memory leak as it would have surfaced after processing so many (60K messages of 1MB).

Answer 1

That looks perfectly fine. Why do you think this is a problem?

The garbage collector will eventually release the unused memory, but this doesn't mean this is going to happen as soon as your service is idle.

Raymond Chen has written a good article explaining the basic idea of garbage collection:

Everybody thinks about garbage collection the wrong way

However - but this is pure speculation with the information given in your questions - there might be memory leaks related to extension methods in your XSLT. Extension methods may lead to problems in case you recompile the stylesheet every time a new XML document is transformed. The fix is simple: Once compiled, cache the stylesheet.

Answer 2

SIR! PUT DOWN TASK MANAGER AND WALK AWAY. Seriously. Memory management in .NET is not tuned for smallest footprint. It is tuned for efficiency. It will hold onto memory rather than release it back to the system. Resist the temptation to babysit your memory unless there is an actual problem (OOM exceptions, system issues).

Answer 3

What measure are you using for memory? There are lots of possible measures, and none of them really mean "used memory". With virtual memory, sharing of images, etc. things are not simple.

even when is idle it has memory of about 500MB

Most processes (and I think this includes .NET) will not release memory back to the OS once allocated to the process. But if it is not in use it will be paged from physical memory allowing other processes to make use of it.

started processing messages of 12 MB in size

If an XML document is expanded into an object model (e.g. XmlDocument) it needs a lot more memory (lots of links to other nodes). Either look at a different model (both XDocument and XPathDocument are lighter weight) or, better, process the XML with XmlReader and thus never have a fully expanded object model.

Answer 4

First of all, add something to your system to allow you to manually invoke garbage collection for the purpose of investigation. The CLR will not necessarily return memory to the O/S unless it is in short supply. You should use this mechanism to manually invoke garbage collection when you have the system in a quiescent state so that you can see if the memory drops back down. (You should invoke GC.Collect(2) twice to ensure objects with finalizers are actually collected and not just finalized.)

If the memory goes down to the quiescent level after a full GC then you do not have a problem: simply that .NET is not being proactive at deallocating memory.

If the memory does not go down, then you have some kind of a leak. As your messages are large, it means their text representations are likely ending up on the Large Object Heap (LOH). This particular heap is not compacted which means that it is easier to leak this memory than with the other heaps.

One thing to watch out for is string interning: if you are manually interning strings this can cause LOH fragmentation.

Answer 5

Very hard to say what may be the problem. I've had success using Ants Memory Profiler when investigating memory issues.

Answer 6

Contrary to the thought I don't think there is a memory leak; I would profile the memory.

Answer 7

I would check your event handlers. If you're not careful detaching those when you're done, it seems like it's easy to create object references that won't be collected by GC.
I don't know that it's the greatest practice (since the onus of beginning and cancelling event handling should fall to the subscriber), but I've gone the route of implementing IDisposable and creating my events with an explicit delegate field instance. On disposal, the field can be set to null, which effectively detaches all subscriptions.

something like this:

public class Publisher
    : IDisposable
{
    private EventHandler _somethingHappened;
    public event EventHandler SomethingHappened
    {
        add { _somethingHappened += value; }
        remove { _somethingHappened -= value; }
    }
    protected void OnSomethingHappened(object sender, EventArgs e)
    {
        if (_somethingHappened != null)
            _somethingHappened(sender, e);
    }

    public void Dispose()
    {
        _somethingHappened = null;
    }
}

Barring that (I don't know how much control you have over the publisher), you might need to be careful to detach unneeded handlers on your subscriber:

public class Subscriber
    : IDisposable
{ 
    private Publisher _publisher;
    public Publisher Publisher
    {
        get { return _publisher; }
        set {
            // Detach from the old reference
            DetachEvents(_publisher);
            _publisher = value;
            // Attach to the new
            AttachEvents(_publisher);
        }
    }

    private void DetachEvents(Publisher publisher)
    {
        if (publisher != null)
        {
            publisher.SomethingHappened -= new EventHandler(publisher_SomethingHappened);
        }
    }
    private void AttachEvents(Publisher publisher)
    {
        if (publisher != null)
        {
            publisher.SomethingHappened += new EventHandler(publisher_SomethingHappened);
        }
    }

    void publisher_SomethingHappened(object sender, EventArgs e)
    {
        // DO STUFF
    }

    public void Dispose()
    {
        // Detach from reference
        DetachEvents(Publisher);
    }
}