5

I have created a sample C# console application, which reads a file data in byte array and converts the byte array to hex string. It requires huge memory and it does not frees the memory after the work is done, I am also nullifying the variables in use.

Here is the sample code: 

string filename = @"F:\\AVSEQ09.DAT"; //file size is 32 MB
string hexData = null;

byte[] fileDataContent = File.ReadAllBytes(filename);
if (fileDataContent != null)
    hexData = BitConverter.ToString(fileDataContent);
fileDataContent = null;
hexData = null;

//GC.Collect();
Console.ReadKey();

If I run this code it takes 433 MB of private working set and if I uncomment the GC.collect call the memmory comes down to 6 MB. Why do I have to call GC.collect explicitly, is it bad to call GC.collect explicitly, how can I free the memmory (to 6 MB) without calling GC.collect?

J. Steen
  • 15,470
  • 15
  • 56
  • 63
Venkat Prabhakar
  • 109
  • 1
  • 5

6 Answers6

9

Garbage collection is the simulation of infinite memory on a machine with finite memory, by means of recycling memory that a valid program can't notice is missing.

The above is a very important notion because, among others, it highlights the fact that the garbage collector doesn't have to do anything as long as it can provide memory every time your program asks for it.

It may not be the most controllable memory management system, but if your program ends up in a situation under memory pressure, the CLR will usually start a garbage collection cycle automatically to relieve some of that pressure. When the system doesn't seem to be under pressure, the collection is postponed, to avoid frequent unnecessary pauses.

Theodoros Chatzigiannakis
  • 28,773
  • 8
  • 68
  • 104
  • 1
    What about when another program needs a large chunk of memory? It seems irresponsible to leave large chunks of memory under the ownership of a program that is no longer using it – Assimilater Aug 11 '17 at 19:15
  • 1
    And I mean large chunks of memory. For my own testing I made a window with a private variable byte[] of size 500,000,000 (6 windows or so causes the system to run out of memory) The garbage collector will leave that memory in ownership of my program until my program asks for more memory long after those windows are gone while my program is basically idle – Assimilater Aug 11 '17 at 19:16
  • 1
    @Assimilater While your concerns are absolutely valid, that's just how tracing GC works. I've personally rolled my own array class in C# that I use for extremely large arrays (think up to 32 GB) that utilizes OS allocators directly and have to option of being "freed" (resized to zero) explicitly through the `IDisposable` interface. – Theodoros Chatzigiannakis Aug 11 '17 at 19:23
  • In the case of a large array that works, and it was simply my test code to help me understand what was going on with the GC. In practice large memory footprints come from other sources in my projects. IO streams that have been disposed (using using statements, not explicitly as private variables), buffers that they have used long since haven fallen out of scope, oxyplot models having fallen out of scope, buffers used to store time and fft data for those plots. Adding a disposable interface to all these classes (models) to compensate for this shortcoming of the GC seems a little much to me... – Assimilater Aug 11 '17 at 19:28
  • I found [this gem](https://stackoverflow.com/questions/867114/why-no-reference-counting-garbage-collection-in-c), will ponder over it later – Assimilater Aug 11 '17 at 20:18
4

You don't have to call GC.Collect explicitly.

It is true that your code uses a lot of memory, however garbage collection has an intelligent algorithm that determines when it should run. It runs when you are out of memory or when you have a sudden increase in memory usage. As long as there is additional memory left, the garbage collector doesn't do anything. This greatly improves performance. If the GC would run constantly your performance would be terrible.

Setting a variable to null is also not necessary in .NET. The runtime will keep track of which variables are in use and will mark them for collection whenever necessary.

Wouter de Kort
  • 39,090
  • 12
  • 84
  • 103
  • +1 for the note on nulls. However, it doesn't just run when you are out of memory, it also runs when memory usage is growing past certain levels that adapt over the lifetime of the process (such as sudden massive memory growth). – Adam Houldsworth Apr 22 '13 at 08:24
2

You shouldn't use GC explicitly, but rather refrain from reading ALL your data in at once. This is rather bad practice you are performing... When reading from files you should use using-statement and read in small buffers.

using(StreamReader sr = new StreamReader(filename)){
     //while has data
     char[] buffer = new char[64000]; // or whatever you like
     sr.ReadBlock(buffer, 0, buffer.Length);     
     //do your conversion here
}

Look here for StreamReader.ReadBlock().

bash.d
  • 13,029
  • 3
  • 29
  • 42
  • 2
    Possibly because it doesn't really address the question. Also, sometimes all the data needs to be read in at once if it can't be restructured to fit a stream pattern. Just FYI, I'm not the downvoter. – Adam Houldsworth Apr 22 '13 at 08:23
1

The .NET CLR manages memory itself, and it calls the garbage collector whenever necessary.

Whatever the case, even where you have time-critical code, you can rely on the .NET CLR calling the Garbage Collector at the appropriate time. The GC is very efficient and frees memory incredibly fast.

So there really is no problem. Do not call the GC, you don't need to. This is .NET.

Roy Dictus
  • 32,551
  • 8
  • 60
  • 76
1

When you deal with such big files and memory structores consider using streams instead of opening and converting the full file in memory. Anyway you should not call GC collect.

Stefano Altieri
  • 4,550
  • 1
  • 24
  • 41
1

The .NET garbage collector is a generational collector. From a generation point of view, large objects (85K or larger) belong to generation 2 because they are collected only when there is a generation 2 collection, which includes all generation, this does not happen as often as generation 0 collection because fo the objects size, large objects are usually arrays (in your case its the byte[]). Generation 2 collections can be caused for example by:

  • Allocation Exceeds the Generation 0 or Large Object Threshold Most GCs happen because of allocations on the managed heap
  • System Is in Low Memory Situation This happens when I receive the high memory notification from the OS.
  • System.GC.Collect Is Called When someone calls GC.Collect on generation 2 (by passing either no arguments to GC.Collect or passing GC.MaxGeneration as an argument)

So when your system will need the memory it will normaly free the byte array without you forcing it to free it by calling GC.Collect.

Take a look at this article

CloudyMarble
  • 36,908
  • 70
  • 97
  • 130