I'm getting a very strange behaviour in my application resulting in OutOfMemoryException. I din't find any clue in the several other posts I found on this matter (e.g., 1, 2, 3).
I have my app compiled in 64bit, my system has 16GB ram and when doing the following tests I had always around 8-10GB of free memory.
I have a version of this app (A) that works perfectly and allocates regularly a total of 5GB of data. Then in the last weeks I've worked to a new version (B). When I try to run or to debug version B, I consistently get OutOfMemoryException every time my process (or the host one) is about to exceed 2GB.
Please note that I'm not using a single object larger than 2GB (even if I've tried also to include gcAllowVeryLargeObjects
in app.config).
The most notably difference with respect to memory usage is that version B allocates in a row a quite large number of List[] arrays (here is where I get the exception). More precisely:
List<T>[] lists = new List<T>[N];
for (int i = 0; i < n; i++)
lists [i] = new List<T>(S);
With:
N=16k
S~=20k
typeof(T)=Int64
This means that I'm allocating about 16k arrays, each one of 160KB (total 2.5GB ram). So initially I thought that maybe the problem was memory fragmentation and that it was not possible to find 16k holes of 160KB contiguous memory each. But what is not clear at all to me is that on Task Manager my process never exceed 2GB when there are still 6GB or more of free memory. Can be the memory so fragmented? Can someone explain this?
UPDATE
I found another block of ram and now my system has 20GB. I do not get anymore the problem. The total memory taken by my process is 4GB. Still I do not understand what I was observing before with 16GB installed memory and with my process not exceeding 2GB...
UPDATE 2
I went back to 16GB of ram and I do not have the OutOfMemoryException. I do not think the installed memory was the problem. Between the installation of memory I went into the Project > Build properties to check the target Platform (see comments from Hans Passant). I verified the platform was x64 and I never changed this. I have the sensation that VS was targeting in 32bit for some reason or bug and that by going to the settings, the correct platform is now targeted. All the story is still obscure to me, but this is the only thing I'm able to guess.