2

I am working on a .NET application for processing geographic data that uses ESRI's ArcObjects COM library via ESRI's own .NET interop assemblies.

When running in production, the process can crash during some operations due to reaching the 2GB per-process memory limit. (ArcObjects is a 32-bit library.) This is because some of the processing steps can create many temporary ArcObjects geometry objects. It will leak memory and eventually run out of memory despite manually releasing these objects using FinalReleaseComObject and associated helper methods. However, I can force the GC to release the memory by calling WaitForPendingFinalizers, and calling it regularly along with GC.Collect and FinalReleaseComObject keeps the memory usage under control. Otherwise, many objects remain in memory until the process exits (normally or abnormally).

First question: why isn't the memory held by the ArcObjects COM objects released immediately? Or, why does the GC allow the process to crash instead of finalizing the released COM objects and reclaiming the memory before it crashes?

The application runs in production on Windows 2008 64-bit, while I develop using Windows 7 32-bit. I can get the process to crash on the production boxes, but not on my development box. I thought that this may have been because locally I usually run in Visual Studio with a Debug build, but I have also tried it without the debugger (Start Without Debugging) using a Release build but even then it didn't use anywhere near as much memory as in production and won't crash.

Second question: Why?

EDIT: In my previous experiments, I worked out that GC.Collect by itself is not sufficient, even though I was calling it explicitly. I have a utility method that calls GC.Collect followed by GC.WaitForPendingFinalizers and calling it after every algorithm iteration keeps memory usage down.

Gnat
  • 2,861
  • 1
  • 21
  • 30

1 Answers1

5

COM objects used in managed applications are sitting behind a runtime-callable wrapper (RCW) which is a managed object that replicates the interface of the COM object towards the managed client (it's a bridge between the managed and unmanaged components). If I remember correctly, the actual COM interface references are held by the RCW, not by your code. When you release the COM object, it's really the RCW that's released but that, being a managed object itself, doesn't go away until the GC gets around to clean it up. When that happens, the RCW is deleted and the last reference to the COM object is gone, so it can destroy itself. (According to the documentation, FinalReleaseComObject should set the refernce count to 0 but I've seen similar behavior in the past so I question whether the documentation is correct.)

As for your second question, I have a guess: I've seen in stressed environments that the GC simply doesn't get a chance to run when the system is under heavy load. Back then we determined that the GC ran on a lower-priority thread and our application was using so much CPU that the GC never had a chance to clean up. We had to add another thread that was periodically calling GC.Collect() every once in a while - this forced the GC thread to awaken and do its magic. You may be facing something similar. (This may be the reason behind the your issue in #1, too.)

xxbbcc
  • 16,930
  • 5
  • 50
  • 83
  • I do know about RCWs and so on, but the memory held by the COM object is not being released despite the GC being run explicitly. I've updated the question. – Gnat Oct 08 '12 at 02:37
  • @Gnat The reason you need `GC.WaitForPendingFinalizers()` is because `GC.Collect()` doesn't actually run the GC. It asks the GC to perform a collection cycle but the GC is running on a different thread and may not get scheduled under heavy load. `GC.WaitForPendingFinalizers()` forces a wait until the end of the next collection cycle - this frees up CPU and the GC has a chance to run and finish memory cleanup. – xxbbcc Oct 08 '12 at 02:41
  • @Gnat if you're asking if it's correct (or expected) that you need to call `GC.WaitForPendingFinalizers()` then yes, it's expected if your application puts a load on the CPU. In high-load, high-memory scenarios the GC thread may get starved and the system (or process) may run out of memory. I had to do the same in that old project I mentioned. – xxbbcc Oct 08 '12 at 02:43
  • I've done some digging and it seems that you may be right. The process is running in workstation GC mode (at least locally), and http://msdn.microsoft.com/en-us/library/ee787088.aspx#workstation_and_server_garbage_collection says that the GC thread must compete for CPU and threads with native code (i.e. most of what the application is doing) are not suspended. Various resources (e.g. http://msdn.microsoft.com/en-us/library/ff647812.aspx, http://stackoverflow.com/questions/12265598) say it's a bad idea to wait for finalizers but I can't see a way around it. – Gnat Oct 08 '12 at 04:25
  • Accepted as answer due to explanatory comments – Gnat Oct 08 '12 at 04:27