1

Some background: We are running some pipelines on a buildserver and it consumes way to much memory. The pipeline does some DB imports and it builds up memory over time x times greater than the total size of an exported DB. For the import Entity Framework (core) is used (in order to be able to reuse entity definitions used in other parts of the application).

Situtation: We are looking into where memory consumption can be reduced. Hence I was using the memory profiler.

I've noticed that sometimes the garbage collector does seem to free up memory after process X was done, and before process Y was started. enter image description here

This is as expected. The 4GB memory build up is OK(ish), as long as it is released. The code that caused this consumption is running in its own Scope (speaking about dependency injection) and the DbContexts (and other things) used are registered as Scoped. Hence we have these ScopeWorkers.

    await _scopeWorker.DoWork<MyProcessX>(_ => _.Import(cancellationToken));
    // In some test, memory got freed up in between, but in some other test, memory never seemed to have dropped
    await _scopeWorker.DoWork<MyProcessY>(_ => _.Import(cancellationToken));

But in some other test, this drop in memory was never seen. The red arrow indicates approximately the same moment in time, after MyProcessX.Import, and a significant drop (of 4GBs) was never seen. enter image description here

Of course I do not know whether the GC spread out the cleaning of this memory over a couple dozen collection moments, instead of 3, as seen in the first screenshot.

Questions

  • Is it possible to wait for the garbage collector to have collected basically all memory used by MyProcessX.Import, before continueing with MyProcessY.Import?
  • Should the garbage collector behave consistently? In other words, should I see the same memory consumption graph over time when the processes is repeated and is doing the exact same operations (so same data, as the data comes from a static source)
  • If the garbage collector is inconsistent in its behavior, how to make good use of the memory profiling feature in Visual Studio to spot opportunities of lowering memory?

EDIT Yes the memory pressure on the system changes everything, as Evk pointed out. After reserving almost all physical memory on the system (31GB/32GB) and continuing the process which I was attempting to optimize memory usage I could see a definite drop in memory used. I could repeat this, as shown in the image there are actually 2 drops in memory. enter image description here

Mike de Klerk
  • 11,906
  • 8
  • 54
  • 76
  • Does this answer your question? [When does garbage collection get triggered in C#?](https://stackoverflow.com/questions/28461652/when-does-garbage-collection-get-triggered-in-c) – Charlieface Oct 24 '22 at 13:55

1 Answers1

2

Garbage collector uses the following conditions to decide whether it should start collection:

The system has low physical memory. The memory size is detected by either the low memory notification from the operating system or low memory as indicated by the host.

The memory that's used by allocated objects on the managed heap surpasses an acceptable threshold. This threshold is continuously adjusted as the process runs.

The GC.Collect method is called. In almost all cases, you don't have to call this method because the garbage collector runs continuously. This method is primarily used for unique situations and testing.

The first point means it depends on all processes running on current machine, not only on your process. For the same reason you don't know when GC will start, so you can't wait for that to happen.

For that same reason it cannot behave consistently in way you describe, in relation to your process. Your process may do the same thing, but OS as a whole is unlikely to ever do the same things during your process run. In one test run there were enough free memory over whole system, and in another it was not.

What you can do is force GC to run via GC.Collect (and overloads). However that's rarely a good idea.

Main thing you should ask yourself is - does high memory consumption bring any problems? Because by itself it's not a problem (assuming no memory leaks) - you have RAM to be used, not to just stay "free". If there is enough memory currently - GC might rightfully decide to not waste time on garbage collection and do that later when necessary.

Evk
  • 98,527
  • 8
  • 141
  • 191