C# .NET Garbage Collection not functioning?

Question

I am working on a relatively large solution in Visual Studio 2010. It has various projects, one of them being an XNA Game-project, and another one being an ASP.NET MVC 2-project.

With both projects I am facing the same issue: After starting them in debug mode, memory usage keeps rising. They start at 40 and 100MB memory usage respectively, but both climb to 1.5GB relatively quickly (10 and 30 minutes respectively). After that it would sometimes drop back down to close to the initial usage, and other times it would just throw OutOfMemoryExceptions.

Of course this would indicate severe memory leaks, so that is where I initially tried to spot the problem. After searching for leaks unsuccesfully, I tried calling GC.Collect() regularly (about once per 10 seconds). After introducing this "hack", memory usage stayed at 45 and 120MB respectively for 24 hours (until I stopped testing).

.NET's garbage collection is supposed to be "very good", but I can't help suspecting that it just doesn't do its job. I have used CLR Profiler in an attempt to troubleshoot the issue, and it showed that the XNA project seemed to have saved a lot of byte arrays I was indeed using, but to which the references should already be deleted, and therefore collected by the garbage collector.

Again, when I call GC.Collect() regularly, the memory usage issues seem to have gone. Does anyone know what could be causing this high memory usage? Is it possibly related to running in Debug mode?

Answer 1

After searching for leaks unsuccesfully

Try harder =)

Memory leaks in a managed language can be tricky to track down. I have had good experiences with the Redgate ANTS Memory Profiler. It's not free, but they give you a 14 day, full-featured trial. It has a nice UI and shows you where you memory is allocated and why these objects are being kept in memory.

As Alex says, event handlers are a very common source of memory leaks in a .NET app. Consider this:

public static class SomeStaticClass
{
    public event EventHandler SomeEvent;
}

private class Foo
{
    public Foo()
    {
        SomeStaticClass.SomeEvent += MyHandler;
    }

    private void MyHandler( object sender, EventArgs ) { /* whatever */ }
}

I used a static class to make the problem as obvious as possible here. Let's say that, during the life of your application, many Foo objects are created. Each Foo subscribes to the SomeEvent event of the static class.

The Foo objects may fall out of scope at one time or another, but the static class maintains a reference to each one via the event handler delegate. Thus, they are kept alive indefinitely. In this case, the event handler simply needs to be "unhooked".

...the XNA project seemed to have saved a lot of byte arrays I was indeed using...

You may be running into fragmentation in the LOH. If you are allocating large objects very frequently they may be causing the problem. The total size of these objects may be much smaller than the total memory allocated to the runtime, but due to fragmentation there is a lot of unused memory allocated to your application.

The profiler I linked to above will tell you if this is a problem. If it is, you will likely be able to track it down to an object leak somewhere. I just fixed a problem in my app showing the same behavior and it was due to a MemoryStream not releasing its internal byte[] even after calling Dispose() on it. Wrapping the stream in a dummy stream and nulling it out fixed the problem.

Also, stating the obvious, make sure to Dispose() of your objects that implement IDisposable. There may be native resources lying around. Again, a good profiler will catch this.

My suggestion; it's not the GC, the problem is in your app. Use a profiler, get your app in a high memory consumption state, take a memory snapshot and start analyzing.

Answer 2

First and foremost, the GC works, and works well. There's no bug in it that you have just discovered.

Now that we've gotten that out of the way, some thoughts:

1. Are you using too many threads?
2. Remember that GC is non deterministic; it'll run whenever it thinks it needs to run (even if you call GC.Collect().
3. Are you sure all your references are going out of scope?
4. What are you loading into memory in the first place? Large images? Large text files?

Your profiler should tell you what's using so much memory. Start cutting at the biggest culprits as much as you can.

Also, calling GC.Collect() every X seconds is a bad idea and will unlikely solve your real problem.

Answer 3

Analyzing memory issues in .NET is not a trivial task and you definitely should read several good articles and try different tools to achieve the result. I end up with the following article after investigations: http://www.alexatnet.com/content/net-memory-management-and-garbage-collector You can also try to read some of Jeffrey Richter's articles, like this one: http://msdn.microsoft.com/en-us/magazine/bb985010.aspx

From my experience, there are two most common reasons for Out-Of-Memory issue:

1. Event handlers - they may hold the object even when no other objects referencing it. So ideally you need to unsubscribe event handlers to destroy the object automatically.
2. Finalizer thread is blocked by some other thread in STA mode. For example, when STA thread does a lot of work the other threads are stopped and the objects that are in the finalization queue cannot be destroyed.

Answer 4

Edit: Added link to Large Object Heap fragmentation.

Edit: Since it looks like it is a problem with allocating and throwing away the Textures, can you use Texture2D.SetData to reuse the large byte[]s?

First, you need to figure out whether it is managed or unmanaged memory that is leaking.

1. Use perfmon to see what happens to your process '.net memory# Bytes in all Heaps' and Process\Private Bytes. Compare the numbers and the memory rises. If the rise in Private bytes outpaces the rise in heap memory, then it's unmanaged memory growth.

2. Unmanaged memory growth would point to objects that are not being disposed (but eventually collected when their finalizer executes).

3. If it's managed memory growth, then we'll need to see which generation/LOH (there are also performance counters for each generation of heap bytes).

4. If it's Large Object Heap bytes, you'll want to reconsider the use and throwing away of large byte arrays. Perhaps the byte arrays can be re-used instead of discarded. Also, consider allocating large byte arrays that are powers of 2. This way, when disposed, you'll leave a large "hole" in the large object heap that can be filled by another object of the same size.

5. A final concern is pinned memory, but I don't have any advice for you on this because I haven't ever messed with it.

Answer 5

I would also add that if you are doing any file access, make sure that you are closing and/or disposing of any Readers or Writers. You should have a matching 1-1 between opening any file and closing it.

Also, I usually use the using clause for resources, like a Sql Connection:

using (var connection = new SqlConnection())
{
  // Do sql connection work in here.
}

Are you implementing IDisposable on any objects and possibly doing something custom that is causing any issues? I would double check all of your IDisposable code.

Answer 6

The GC doesn't take into account the unmanaged heap. If you are creating lots of objects that are merely wrappers in C# to larger unmanaged memory then your memory is being devoured but the GC can't make rational decisions based on this as it only see the managed heap.

You end up in a situation where the GC collector doesn't think you are short of memory because most of the things on your gen 1 heap are 8 byte references where in actual fact they are like icebergs at sea. Most of the memory is below!

You can make use of these GC calls:

System::GC::AddMemoryPressure(sizeOfField);
System::GC::RemoveMemoryPressure(sizeOfField);

These methods allow the garbage collector to see the unmanaged memory (if you provide it the right figures)