32

I am testing how big a collection could be in .Net. Technically, any collection object could grows to the size of the physical memory.

Then I tested the following code in a sever, which has 16GB memory, running Windows 2003 server and Visual Studio 2008. I tested both F# and C# code, and looked at the Task Manager while running. I can see that after about growing 2GB memory, the program crashed with out-of-memory exception. I did set the target platform to x64 in the property page. 

open System.Collections.Generic

let d = new Dictionary<int, int>()

for i=1 to 1000000000 do
    d.Add(i,i)

I did a same test to the C5 collection library. The result is that the dictionary in C5 could use up the whole memory. The code uses C5:

let d = C5.HashDictionary<int, int> ()
for i=1 to 1000000000 do
    d.Add(i,i)

Anyone knows why?

Guy Coder
  • 24,501
  • 8
  • 71
  • 136
Yin Zhu
  • 16,980
  • 13
  • 75
  • 117
  • 12
    Physical memory has nothing whatsoever to do with maximum sizes. Windows has used virtual memory management for over a decade now. You can have objects that are far, far larger than available physical memory because of course available physical memory could be *zero* when a new process is created. That's the whole *point* of virtual memory; that physical memory is just a performance optimization of virtual memory. (If there is not enough physical memory to contain the working set then performance degrades, but everything should still *work*.) – Eric Lippert Sep 07 '10 at 15:10
  • 2
    @EricLippert Understood and agreed but what about the reverse... Should you ever get an OOM when there's physical memory available? (assuming it's a real OOM not a too many handles/similar issue). In this instance, it looks like a CLR-imposed limit was the issue but that only means the OPs question is a good one in my opinion – Basic Jul 05 '12 at 09:28

4 Answers4

42

The Microsoft CLR has a 2GB maximum object size limit, even the 64 bit version. (I'm not sure whether this limit is also present in other implementations such as Mono.)

The limitation applies to each single object -- not the total size of all objects -- which means that it's relatively easy to workaround using a composite collection of some sort.

There's a discussion and some example code here...

There seems to be very little official documentation that refers to this limit. It is, after all, just an implementation detail of the current CLR. The only mention that I'm aware of is on this page:

When you run a 64-bit managed application on a 64-bit Windows operating system, you can create an object of no more than 2 gigabytes (GB).

LukeH
  • 263,068
  • 57
  • 365
  • 409
22

In versions of .NET prior to 4.5, the maximum object size is 2GB. From 4.5 onwards you can allocate larger objects if gcAllowVeryLargeObjects is enabled. Note that the limit for string is not affected, but "arrays" should cover "lists" too, since lists are backed by arrays.

Marc Gravell
  • 1,026,079
  • 266
  • 2,566
  • 2,900
11

And to be clear, a Dictionary uses a single array to add the pairs. It is grown (doubled?) each time it is full. When there are 512 million objects, its size is 2GByte (with a 32 bit object pointer, and assuming perfect distribution). Adding one more element makes the Dictionary try to double the array size again. Boom.

The C5 HashDictionary uses linear hashing, and probably uses an array of buckets each containing multiple (16?) elements. It should run into the same problem (much) later.

Stephan Eggermont
  • 15,847
  • 1
  • 38
  • 65
1

The "allow large objects" will only help to get rid of OOM exception.

When one needs to store very many objects the problem that you will see is GC stalls(pauses). What we have done is "hiding" of data from GC, which turned into a very practical solution.

See this: https://www.infoq.com/articles/Big-Memory-Part-3

You can use cache that works as a dictionary: https://github.com/aumcode/nfx/tree/master/Source/NFX/ApplicationModel/Pile

see the caching section

itadapter DKh
  • 596
  • 3
  • 7