Why does my program's memory usage rise quickly and then go flat?

Question

Hi everyone I'm making a C# WinForms application that searches for duplicate images in a directory. It starts by calling a constructor with every image in the directory.

There is a lot of files in the directory and memory was quickly rising to 2gb and then the program would throw an out of memory exception.

Now I've added a check in my for loop to check if the memory has exceeded 800 Megabits and I force a garbage collection. But I've noticed after the first forced collection the memory no longer rises. (The forced garbage collection occurs at loop ~180 out of ~800 then it never occurs again)

(It looks a little like a shark fin swimming through the water leaving waves in its wake.)

I'm stumped as to why this is happening and have come here in search of help.

private void GeneratePrints()
{
    for (int i = 0; i < files.Count; i++)
    {
        if (imageFileExtensions.Contains(Path.GetExtension(files[i])))
            prints.Add(new FilePrint(directory + "/" + files[i]));

        //800 Megabits
        const long MAX_GARBAGE = 800 * 125000;

        if (GC.GetTotalMemory(false) > MAX_GARBAGE)
        {
            GC.Collect();
            GC.WaitForPendingFinalizers();
        }
    }

    Console.WriteLine("File Prints Complete.");
}

GeneratePrints() is called 1 time, once a directory is selected.

I will also show you the constructor for the FilePrint class. I'm pretty sure this all has something to do with the MemoryStream object.

public FilePrint(string filename)
{
    Bitmap img;

    using (var fs = new FileStream(filename, FileMode.Open))
    {
        using (var ms = new MemoryStream())
        {
            fs.CopyTo(ms);
            ms.Position = 0;
            img = (Bitmap)Bitmap.FromStream(ms);

            ms.Close();
        }

        fs.Close();
    }

    this.size = img.Size;
    img = ResizeImage(img, 8, 8);
    img = MakeGrayscale(img);

    //I do some basic for-loop arithmetic here 
    //calculating the average color of the image, not worth posting.

    img.Dispose();
}

So basically I'm wondering how can I make it so that the 'shark-fin-like' memory usage spike at the start never happens so that I do not have to force a garbage collection.

Here is a memory snapshot I have taken when the forced garbage collection occurs (Am I not disposing of the MemoryStreams properly?):

Thank you for your replies, ideas and answers in advance!

[Maybe related](https://stackoverflow.com/questions/12709360/whats-the-difference-between-bitmap-clone-and-new-bitmapbitmap) — ProgrammingLlama, Jul 03 '19 at 04:43
Stupid remark, but why do you go through the trouble of opening two different streams, where you could use `using (Bitmap img = new Bitmap(filename))` — Antoine, Jul 03 '19 at 06:07
I have no idea, why it doesn't build up again, but I do see that you are doing way too much in that ctor. Most of it should be in a method. Also, after the dispose, try setting `img=null;` and leave the GC alone. — Fildor, Jul 03 '19 at 06:22
Unrelated: `directory + "/" + files[i]` - I recommend `IO.Path.Combine(directory, files[i])`. Spares you some headaches. — Fildor, Jul 03 '19 at 06:27
Hi @Antoine, I do this because I modify filenames/delete the files I am working with, I'm pretty sure I don't use that method because when I first started this project I tried that and it locked the file from modification, the method I am using does not, it effectively loads a copy of the image I think. But I in this place I may be able to use your method as I won't be modifying it, just some recycled code that hasn't yet been optimised. — Issung, Jul 03 '19 at 06:33
@Antoine I tried this and it really helped, I think it was because the MemoryStreams were taking up so much memory for some reason, getting rid of the MemoryStreams has really cut down the memory usage and now my foced garbage collection no longer occurs, thank you! Still raises the question of why it was happening though.. — Issung, Jul 03 '19 at 10:19

dbc · Answer 1 · 2019-07-03T07:04:39.703

You don't show the methods ResizeImage(img, 8, 8) and MakeGrayscale(img), but most likely they simply create and return a new image based on the old. If that's true, your code constructs two Bitmap objects that it never explicitly disposes, so try disposing them e.g. as follows:

using (var old = img)
    img = ResizeImage(old, 8, 8);
using (var old = img)
    img = MakeGrayscale(old);

You might also want to guarantee that the final img is disposed using a try/finally:

Bitmap img = null;
try
{
    img = new Bitmap(filename);  // Here I simplified the code, but this will leave the file locked until `img` is disposed after resizing.

    this.size = img.Size;
    using (var old = img)
        img = ResizeImage(old, 8, 8);
    using (var old = img)
        img = MakeGrayscale(old);

    //I do some basic for-loop arithmetic here 
    //calculating the average color of the image, not worth posting.
}
finally
{
    if (img != null)
        img.Dispose();
}

The possible reason you get the long buildup in memory use then a precipitous drop is that eventually the unmanaged resources of the undisposed images will get finalized, however because the GC is unaware of unmanaged memory owned by the undisposed Bitmap objects, it doesn't necessarily kick in, identify the bitmaps as unreferenced and pass them on to the finalizer thread for quite a while. It's not always easy to predict when or even if the finalizer thread will spin up and start working; see Are .net finalizers always executed? to which the answer is not necessarily. But by calling GC.WaitForPendingFinalizers(); you may be kickstarting that process.

Incidentally, a MemoryStream doesn't actually need to be disposed in the current implementation as it lacks unmanaged resources. See this answer by Jon Skeet to Is a memory leak created if a MemoryStream in .NET is not closed? for confirmation. (It's still good practice to do so, though in the case of Bitmap there's that pesky file/stream lock that makes it impossible.)

Why does my program's memory usage rise quickly and then go flat?

1 Answers1