1

I am launching many threads simultaneously, each one writing / reading data into/from a queue; Data is dequeued progressively while it is processed and stored to DB. For some reason the memory is not freed although the Queues are empty and I made sure all event subscription between the data reader and the data processor are unsubscribed at the end of the threads.The amount of squatted RAM is Exactly the amount of the data that is read by binary readers and put into queues.

In order to isolate the problem, I have bypassed the Processing and the DB Storing step.
Why would be the RAM still squatted long after all threads are finished, until I explicitly call GC.Collect() or terminate the program? I manually nullified the Queue after it is emptied, and also nullified the Binary Reader that read the data. I thought that would be enough for the GC to wake up and do its housekeeping at least after a few minutes.

enter image description here

EDIT : The (reformulated after deletion) QUESTION : In short, I was always told that the default GC behaviour managed the memory properly and that I shall almost never call the GC explicitly and let the framework do the job.
I would like to know why in this case memory usage drops down only when an explicit Call is made to GC.Collect

EDIT :
Without GC Collect
enter image description here

With GC Collect (Called on a regular basis) enter image description here

Mehdi LAMRANI
  • 11,289
  • 14
  • 88
  • 130
  • Your problem can be related to many things, without taking a look into your code is quite hard predict why GC is not using the broom, if you want we can use VS Anywhere to review the code together... – Jesus Salas Feb 01 '12 at 23:23
  • 1
    Nice graph but where is the problem? – H H Feb 01 '12 at 23:27
  • 2
    You seem like a pretty clever person thats quite animated, but you do have trouble staying on topic. The last part of your post is "The Question: ...." followed by a funny rant with no direct question. Although I think you already nested your question earlier in your post – Abdul Hfuda Feb 01 '12 at 23:28
  • memory is not free after all threads ends.... this is a wrong behavior, I don't think .net wait for memory pressure to execute GC and cleanup objects, if app is idle it should execute and cleanup everything – Jesus Salas Feb 01 '12 at 23:29
  • This graph had nothing (or at least very little) to do with GC... GC puts memory in the free heap, Windows will still see it as allocated (to the App). But like I said, there is no problem here. – H H Feb 01 '12 at 23:32
  • Maybe he can check this article to hook to GC Events http://www.abhisheksur.com/2010/08/garbage-collection-notifications-in-net.html or create a dump file to use sos to check for real object on internal heaps – Jesus Salas Feb 01 '12 at 23:33
  • @HenkHolterman : The question was deleted by editor. probably did not like the way I wrote it. See new edit. – Mehdi LAMRANI Feb 01 '12 at 23:44
  • @HenkHolterman : "This graph had nothing (or at least very little) to do with GC" : Sorry, NOT TRUE, I fully disagree : When I comment the GC.Collect line I NEVER get the CLEAR memory usage drop-down. Proofed and tested... – Mehdi LAMRANI Feb 01 '12 at 23:46
  • @HenkHolterman : If i was as omniscient as you surely are I would not be asking stupid questions. Sorry to bother the community with my ignorance (though this was how I always made progress). I know I will never be as sapient as you are, but I can live with it :) – Mehdi LAMRANI Feb 01 '12 at 23:51
  • 1
    If you want to learn something, start by closing the TaskManager. – H H Feb 01 '12 at 23:58
  • @Xeno : You are totally right and I apologize. It's 1AM and I'm still stuck at the office with this issue, I make digressions when I feel tired. Bad Habit – Mehdi LAMRANI Feb 02 '12 at 00:00
  • @HenkHolterman : You seem to be so adamant : My app is HPC computing oriented and it often saturates the available RAM. I use the TM (but also other robust Specific Process Monitoring Tools) to monitor how my process is consuming memory through time. Why would that be so bad ? – Mehdi LAMRANI Feb 02 '12 at 00:03
  • I don't have the energy to read all of the comments, but a couple of things to note. 1) Task Manager is **not** an acceptable memory profiler, especially not for a managed application. Stop using it and your problem will go away. Invest in a *real* memory profiler if you really think you have a memory leak. 2) **Never** call `GC.Collect()` in a real application. It's not meant for you to call as a programmer. You do not (and *should not*) "help" the garbage collector. – Cody Gray - on strike Feb 02 '12 at 04:12
  • 3) Memory leaks are not a problem if you dispose resources properly when you're finished with them. Wrap the creation of all objects that implement `IDisposable` with a `using` statement, or make sure that you explicitly call their `Dispose` method before letting them go out of scope. – Cody Gray - on strike Feb 02 '12 at 04:13
  • @CodyGray. Thank you for your insights. I have used IDisposable on some of the classes (but not all) with no difference so far. I'll give it a shot again. However, 1 - If a GC Call is a such a bad thing, why is it freeing memory (last screenshot) while it is NOT freed when it is NOT called explicitly. 2 - would you please tell me why the use of Task manager is such a bad Idea? It gives a fair idea of how my app is behaving. I am also using a mem profiler that gives specific process memory usage graphs and am having the same ouput – Mehdi LAMRANI Feb 02 '12 at 11:40
  • I really can't answer all these questions exhaustively in a comment, and there's little point in leaving an answer since you've already accepted one and seem to be convinced of its correctness. I didn't mean that you should implement `IDisposable` on your *own* classes, but rather make sure that you call the `Dispose` method for all of those classes you use that *do* implement it. Invoking the GC explicitly will free memory, yes. That's what it does. But that violates the whole premise of garbage collection. This is a fundamental misunderstanding. You don't need to free the memory. – Cody Gray - on strike Feb 03 '12 at 02:26
  • If it needed to be freed, it would happen automatically. Task Manager is, quite simply, wrong here. Using it is a bad idea because it gives you the wrong idea. It is *not* giving you a fair idea of how your app is behaving because it's not reporting the data you think it is. And yes, the app *does* consume memory. That's what apps do. It's not *leaking* though unless you've forgotten to dispose something. Otherwise, it's just not being freed. **Garbage collection is indeterminate.** Stop trying to make it determinant. You aren't going to beat the system, you'll just shoot yourself in the foot. – Cody Gray - on strike Feb 03 '12 at 02:27

3 Answers3

4

There are three conditions when a Garbage collection might occur (see MSDN):

1.) The system has low physical memory.

2.) The memory that is used by allocated objects on the managed heap surpasses an acceptable threshold. This means that a threshold of acceptable memory usage has been exceeded on the managed heap. This threshold is continuously adjusted as the process runs.

3.) The GC.Collect method is called. In almost all cases, you do not have to call this method, because the garbage collector runs continuously. This method is primarily used for unique situations and testing.

From your description it sounds like the framework decided neither 1.) nor 2.) are the case hence it will only collect when you call GC.Collect()

BrokenGlass
  • 158,293
  • 28
  • 286
  • 335
  • Thank you for your insights. "an acceptable threshold" is a very vague notion. This is not very Memory-efficient because It limits the amount of free RAM that could be used by other threads (scalability). – Mehdi LAMRANI Feb 01 '12 at 23:28
  • 3
    But where does it say that the App will return memory to Windows? GC.Collect() only reorganizes mem inside the heap(s). – H H Feb 01 '12 at 23:29
  • @MikaJacobi: Not really, once other threads do start to allocate more memory eventually 1.) or 2.) will be triggered causing a GC – BrokenGlass Feb 01 '12 at 23:30
1

Based on the screenshot, it appears that your memory usage never goes over 50% during the entire run of the application. It's much faster for the .NET framework to leave that memory alone than to stop your application (or at least take up a lot of CPU time in checking for liveness) in order to collect the garbage. If you cut your machine's RAM to 2 GB or so, I'm sure that the garbage collector would step up and keep memory usage within the hardware limits.

Adam Mihalcin
  • 14,242
  • 4
  • 36
  • 52
1

It seems you are leaking objects you should use windbg to create a dump then use sos.dll to track root for your objects.

You can follow this explanation on how to track "roots" for your objects and see what is causing these leaks.

Jesus Salas
  • 541
  • 3
  • 8
  • Yes could be, I already did that and identified some leaks. But this does not seem to be leak related, as the GC call seems to free the memory, which would not be the case if there was a plain memory leak (well, i guess) (see my older question here : http://stackoverflow.com/questions/5818802/how-to-identify-where-instances-of-an-object-are-still-referenced) – Mehdi LAMRANI Feb 01 '12 at 23:55
  • There could be a possibility that your objects has been moved to a higher generation, what means also less frequency to be collect by GC, attaching windbg and using sos before you run GC.Collect() can tell you on what generation they are and clarify if this is what is happening really, just one test – Jesus Salas Feb 02 '12 at 00:01
  • take a look also on how to use CLR Profiler for [Profiling the .NET Garbage-Collected Heap](http://msdn.microsoft.com/en-gb/magazine/ee309515.aspx) – Jesus Salas Feb 02 '12 at 00:11
  • Some news about windbg / sos output? Maybe your really don't want to look into the real problem for some reason.... – Jesus Salas Feb 02 '12 at 13:13
  • No, not at all. I will sure take time to do it (again). However I was just tackling the GC issue first, in order to understand why I really needed to call it to free the mem in my case. – Mehdi LAMRANI Feb 02 '12 at 13:31
  • Generally Speaking you have a Symptom, it seems you need to call GC.Collect() for GC do work, focusing on what you name a "GC Issue", (GC has not any issue), you are not going to solve your problem, you have some glitch somewhere in your code and you should do a full analisys. If you prefer to be blind... is ok :) – Jesus Salas Feb 02 '12 at 14:09
  • Ok lol. It is not as if I did not do any profiling at all. I am still looking at ways to tackle this. Just trying to do it step by step, and isolate each part to have a precise understanding of each part before getting the big picture. I will do a deeper profiling later. However, This is still very strange as, if there is a leak, i really can't see where (I described all resources Queues/Readers/Threads I am using, and they are 'appearantly' all freed and nullified and disposed. I will triple check again – Mehdi LAMRANI Feb 02 '12 at 15:07