C# Too Much Memory Usage

Question

I have a process going with multiple steps defied. (Let's say generic implementation of a strategy pattern) where all steps are passing a common ProcessParameter object around. (read/write to it)

This ProcessParameter is an object having many arrays and collections. Example:

class ProcessParameter() {
   public List<int> NumbersAllStepsNeed {get; set;}
   public List<int> OhterNumbersAllStepsNeed {get; set;}
   public List<double> SomeOtherData {get; set;}
   public List<string> NeedThisToo {get; set;}
   ...
}

Once the steps finished, I'd like to make sure the memory is freed and not hang around, because this can have a big memory footprint and other processes need to run too.

Do I do that by running:

pParams.NumbersAllStepsNeed = null;
pParams.OhterNumbersAllStepsNeed = null;
pParams.SomeOtherData = null;
...

or should ProcessParameter implement IDosposable, and Dispose method would do that, and then I just need to use pParams.Dispose() (or wrap it in using block)

What is the best and most elegant way to clean the memory footprint of the used data of one process running?

Does having arrays instead of lists change anything? Or Mixed? The actual param type I need is collections/array of custom objects.

Am I looking in the right direction?

UPDATE

Great questions! Thanks for the comments! I used to have this process running as a single run and I could see memory usage go very high and then gradually down to "normal".

The problem came when I started chaining this processes on top of each other with different stating parameters. That is when memory went unreasonably high, so I want to include a cleaning step between two processes and looking for best way to do that.

There is a DB, this params is a sort of "cache" to speed up things.

Good point on IDisposable, I do not keep unmanaged resources in the params object.

Are you trying to fix an actual problem or asking how to implement premature optimization? — Dan Wilson, Aug 23 '18 at 13:44
`IDisposable` is used to free **unmanaged** resources. To make a managed object eligible for GC you just need to make sure that there are no references to it. See also [What is the correct way to free memory in C#](https://stackoverflow.com/q/6066200/1260204) — Igor, Aug 23 '18 at 13:44
Once the `ProcessParameter` is no longer reachable and gets collected, all of the objects contained in its properties will, too. In general, unless you're hanging on to `ProcessParameter` instances for much longer than the sub-objects, there should be no need to explicitly `null` them. If there *is* such a need, you should seriously consider splitting up the object before resorting to semi-manual memory management shenanigans. For starters, such things are `NullReferenceException` accidents waiting to happen. — Jeroen Mostert, Aug 23 '18 at 13:44
I agree with Jeroen. "Using" won't be of any help, I guess inferring from the usage description you give, the lifetime won't be handled inside one "using" block. And I'd like to add that you probably should be careful to not "leak" references, so those big ones can be reclaimed by GC and that's all. You should be fine. — Fildor, Aug 23 '18 at 13:47
If unmanaged resources are not involved, probably GC.Collect and GC.WaitForPendingFinalizers is what are you looking for, and not Dispose. However, such hacks are useless in most cases... — Alex F, Aug 23 '18 at 13:49
But if you _do_ have too much memory in use, maybe think about using a DB? (if applicable) — Fildor, Aug 23 '18 at 13:50
Great questions! Thanks for the comments! I updated the question. — DDan, Aug 23 '18 at 13:59
Can you show us how you're setting your properties? Have you considered Lazy loading? — Parrish Husband, Aug 23 '18 at 14:08
For caches, there are other options to consider: `MemoryCache`, which allows customizable policies and a memory limit, `WeakReference`, which allows garbage collection to reclaim an object despite the weak reference, and `ConditionalWeakTable`, which allows you to "attach" properties to an object that disappear when the object does. These are solutions in search of a problem, though: the first priority is to properly put the application through a memory profiler and see if you can reduce its memory usage by simply allocating less. — Jeroen Mostert, Aug 23 '18 at 14:11
Also, if the main problem is cooperating nicely with other processes on the same machine (i.e., using as much memory as is available, but limiting the actual amount of memory available to the process), you can use a job object or a Docker container to limit the memory used, separate of what you do in the application itself to reduce actual memory use, and let the garbage collector take care of the rest. [For example](https://stackoverflow.com/q/6266820/4137916). — Jeroen Mostert, Aug 23 '18 at 14:15
@JeroenMostert Thank you for the guidance. If you consolidate your comments into an answer it qualifies as an answer to the question. — DDan, Aug 23 '18 at 14:28
I gave this a close vote because it lacks a specific, verifiable example. You're asking a general question about a specific use case. The general case has already been dealt with by the .NET framework, so we need the specifics. — theMayer, Aug 23 '18 at 15:28

score 2 · Answer 1 · answered Aug 23 '18 at 15:21

Whilst using the Disposal pattern is a good idea, I don't think it will give you any extra benefits in terms of freeing up memory.

Two things that might:

Call GC.Collect()

However, I really wouldn't bother (unless perhaps you are getting out of memory exceptions). Calling GC.Collect() explicity may hurt performance and the garbage collector really does do a good job on its own. (But see LOH - below.)

Be aware of the Large Object Heap (LOH)

You mentioned that it uses a "big memory footprint". Be aware that any single memory allocation for 85,000 bytes or above comes from the large object heap (LOH). The LOH doesn't get compacted like the small object heap. This can lead to the LOH becoming fragmented and can result in out of memory errors even when you have plenty of available memory.

When might you stray into the LOH? Any memory allocation of 85,000 bytes or more, so on a 64 bit system that would be any array (or list or dictionary) with 10,625 elements or more, image manipulation, large strings etc.

Three strategies to help minimise fragmentation of the LOH:

i. Redesign to avoid it. Not always practical. But a list of lists or dictionary of dictionaries might avoid the limit. This can make the implementation more complex so I wouldn't unless you really need to, but on the plus side this can be very effective.

ii. Use fixed sizes. If all of more of your memory allocations in the LOH are the same size then this will help minimise any fragmentation. For example for dictionaries and lists set the capacity (which sets the size of the internal array) to the largest size you are likely to use. Not so practical if you are doing image manipulation.

iii. Force the garbage collector to compact the LOH:

System.Runtime.GCSettings.LargeObjectHeapCompactionMode = System.Runtime.GCLargeObjectHeapCompactionMode.CompactOnce;   
GC.Collect();

you do need to be using .NET Framework 4.5.1 or later to use that. This is probably the simplest approach. In my own applications I have a couple of instances where I know I will be straying into the LOH and that fragmentation can be an issue and I set

System.Runtime.GCSettings.LargeObjectHeapCompactionMode = System.Runtime.GCLargeObjectHeapCompactionMode.CompactOnce;

as standard in the destructor - but only call GC.Collect() explicitly if I get an out of memory exception when allocating.

Hope this helps.

C# Too Much Memory Usage

1 Answers1