How do I load many large photos from a directory and its sub-directories in such a way as to prevent an OutOfMemoryException?
I have been using:
foreach(string file in files)
{
PictureBox pic = new PictureBox() { Image = Image.FromFile(file) };
this.Controls.Add(pic);
}
which has worked until now. The photos that I need to work with now are anywhere between 15 and 40MB's each, and there could be hundreds of them.
You're attacking the garbage collector with this approach. Loading 15-40mb objects in a loop will always invite an OutOfMemoryException. This is because the objects go straight onto the large object heap, all objects > 85K do. Large objects become Gen 2 objects immediately and the memory is not automatically compacted as of .Net 4.5.1 (you request it) and will not be compacted at all in earlier versions.
Therefore even if you get away with initially loading the objects and the app keeps running, there is every chance that these objects, even when dereferenced completely, will hang around, fragmenting the large object heap. Once fragmentation occurs and for example the user closes the control to do something else for a minute or two and opens the control again, it is much more likely all the new objects will not be able to slot in to the LOH - the memory must be contiguous when allocation occurs. The GC runs collections on Gen 2 and LOH much less often for performance reasons - memcpy is used by the GC in the background and this is expensive on larger blocks of memory.
Also, the memory consumed will not be released if you have all of these images referenced from a control that is in use as well, imagine tabs. The whole idea of doing this is misconceived. Use thumbnails or load full scale images as needed by the user and be careful with the memory consumed.
UPDATE Rather than telling you what you should and should not do I have decided to try to help you do it :)
I wrote a small program that operates on a directory containing 440 jpeg files with a total size of 335 megabytes. When I first ran your code I got the OutOfMemoryException and the form remained unresponsive.
Step 1 The first thing to note is if you are compiling as x86 or AnyCpu you need to change this to x64. Right click project, go to Build tab and set the target platform to x64.
This is because the amount of memory that can be addressed on a 32 bit x86 platform is limited. All .Net processes run within a virtual address space and the CLR heap size will be whatever the process is allowed by the OS and is not really within the control of the developer. However, it will allocate as much memory as is available - I am running on 64 bit Windows 8.1 so changing the target platform gives me an almost unlimited amount of memory space to use - right up to the limit of physical memory your process will be allowed.
After doing this running your code did not cause an OutOfMemoryException
Step 2
I changed the target framework to 4.5.1 from the default 4.5 in VS 2013. I did this so I could use GCSettings.LargeObjectHeapCompactionMode, as it is only available in 4.5.1 . I noticed that closing the form took an age because the GC was doing a crazy amount of work releasing memory. Basically I would set this at the end of the loadPics code as it will allow the large object heap to not get fragmented on the next blocking garbage collection. This will be essential for your app I believe so if possible try to use this version of the framework. You should test it on earlier versions too to see the difference when interacting with your app.
Step 3 As the app was still unresponsive I made the code run asynchronously
Step 4 As the code now runs on a separate thread to the UI thread it caused a GUI cross thread exception when accessing the form, so I had to use Invoke which posts a message back to the UI thread from the code's thread. This is because UI controls can only be accessed from a UI thread.
Code
private async void button1_Click(object sender, EventArgs e)
{
await LoadAllPics();
}
private async Task LoadAllPics()
{
IEnumerable<string> files = Directory.EnumerateFiles(@"C:\Dropbox\Photos", "*.JPG", SearchOption.AllDirectories);
await Task.Run(() =>
{
foreach(string file in files)
{
Invoke((MethodInvoker)(() =>
{
PictureBox pic = new PictureBox() { Image = Image.FromFile(file) };
this.Controls.Add(pic);
}));
}
}
);
GCSettings.LargeObjectHeapCompactionMode = GCLargeObjectHeapCompactionMode.CompactOnce;
}
You can try resizing the image when you are putting on the UI.
foreach(string file in files)
{
PictureBox pic = new PictureBox() { Image = Image.FromFile(file).resizeImage(50,50) };
this.Controls.Add(pic);
}
public static Image resizeImage(this Image imgToResize, Size size)
{
return (Image)(new Bitmap(imgToResize, size));
}
