Maximum capacity of Collection different than expected for x86

Question

The main question in about the maximum number of items that can be in a collection such as List. I was looking for answers on here but I don't understand the reasoning.

Assume we are working with a List<int> with sizeof(int) = 4 bytes... Everyone seems to be sure that for x64 you can have a maximum 268,435,456 int and for x86 a maximum of 134,217,728 int. Links:

However, when I tested this myself I see that it's not the case for x86. Can anyone point me to where I may be wrong?

//// Test engine set to `x86` for `default processor architecture`
[TestMethod]
public void TestMemory()
{
    var x = new List<int>();

    try
    {
        for (long y = 0; y < long.MaxValue; y++)
        x.Add(0);
    }
    catch (Exception)
    {
        System.Diagnostics.Debug.WriteLine("Actual capacity (int): " + x.Count);
        System.Diagnostics.Debug.WriteLine("Size of objects: " + System.Runtime.InteropServices.Marshal.SizeOf(x.First().GetType())); //// This gives us "4"
    }
}

For x64: 268435456 (expected)

For x86: 67108864 (2 times less than expected)

Why do people say that a List containing 134217728 int is exactly 512MB of memory... when you have 134217728 * sizeof(int) * 8 = 4,294,967,296 = 4GB... what's way more than 2GB limit per process. Whereas 67108864 * sizeof(int) * 8 = 2,147,483,648 = 2GB... which makes sense.

I am using .NET 4.5 on a 64 bit machine running windows 7 8GB RAM. Running my tests in x64 and x86.

EDIT: When I set capacity directly to List<int>(134217728) I get a System.OutOfMemoryException.

EDIT2: Error in my calculations: multiplying by 8 is wrong, indeed MB =/= Mbits. I was computing Mbits. Still 67108864 ints would only be 256MB... which is way smaller than expected.

Have you tried setting the list capacity upfront? Like so: `var x = new List(134217728);`. Remember the internal array is expanded by reallocating a new one... that might spoil the memory assignment. Only a clue, though, I'm far from being an expert. Also, why are you multiplying every quantity by 8 on both of your final calculations? — Leandro, Apr 02 '14 at 16:19
Why are you multiplying the size to 8? you don't need that if you want to ger result in bytes. So the size of list on x64 is exactly 512 MB — Sasha, Apr 02 '14 at 16:19
@LeandroTaset Unfortuantely, I get an `System.OutOfMemoryException` when I set capacity directly. This should expected because we exceed the 2GB memory limit. — Vlad, Apr 02 '14 at 16:22
@OleksandrPshenychnyy Good spot, you are correct, I added an edit. — Vlad, Apr 02 '14 at 16:30
can you replace your writeline by this and show the result? System.Diagnostics.Debug.WriteLine("Actual capacity (int): {0}, {1}", x.Count, System.Runtime.InteropServices.Marshal.SizeOf(x.First().GetType())); — Fredou, Apr 02 '14 at 16:40
@Fredou That results in: `Actual capacity (int): 67108864, 4` — Vlad, Apr 02 '14 at 16:42
@Fredou It's a WPF app and I am running this code inside a `[TestMethod]` with the test engine set to x86. I will give it a try outside the test engine. — Vlad, Apr 02 '14 at 16:46

score 8 · Accepted Answer · answered Apr 02 '14 at 16:49

The underlying storage for a List<T> class is a T[] array. A hard requirement for an array is that the process must be able to allocate a contiguous chunk of memory to store the array.

That's a problem in a 32-bit process. Virtual memory is used for code and data, you allocate from the holes that are left between them. And while a 32-bit process will have 2 gigabytes of memory, you'll never get anywhere near a hole that's close to that size. The biggest hole in the address space you can get, right after you started the program, is around 500 or 600 megabytes. Give or take, it depends a lot on what DLLs get loaded into the process. Not just the CLR, the jitter and the native images of the framework assemblies but also the kind that have nothing to do with managed code. Like anti-malware and the raft of "helpful" utilities that worm themselves into every process like Dropbox and shell extensions. A poorly based one can cut a nice big hole in two small ones.

These holes will also get smaller as the program has been allocating and releasing memory for a while. A general problem called address space fragmentation. A long-running process can fail on a 90 MB allocation, even though there is lots of unused memory laying around.

You can use SysInternals' VMMap utility to get more insight. A copy of Russinovich's book Windows Internals is typically necessary as well to make sense of what you see.

+1 - looking at memory map is the right approach to understand the issue. — Alexei Levenkov, Apr 02 '14 at 16:51
Great answer. I made sure I wasn't loading any extra DLLs, and made sure nothing else is executing BUT that code... Looks like test engine has some extra stuff in it. Now the maximum capacity is indicated as 134,217,728 `int`. Thank you very much, it makes sense. — Vlad, Apr 02 '14 at 16:53

score 1 · Answer 2 · answered Apr 02 '14 at 16:57

1

This could maybe also help but i was able to replicate this 67108864 limit by creating a test project with the provided code

in console, winform, wpf, i was able to get the 134217728 limit

in asp.net i was getting 33554432 limit

so in one of your comment you said [TestMethod], this seem to be the issue.

answered Apr 02 '14 at 16:57

Fredou

19,848
10
58
113

Indeed, that does seem to be the issue. Hans's answer led me to this conclusion as well. – Vlad Apr 02 '14 at 16:58

AnthonyLambert · Answer 3 · 2014-04-02T16:25:28.383

0

While you can have MaxValue Items, in practice you will run out of memory before then.

Running as x86 the most ram you can have even on a x46 box would be 4GB more likely 2GB or 3GB is the max if on a x86 version of Windows.

The available ram is most likely much smaller as you would only be able to allocate the biggest continuous space to the array.

edited Apr 02 '14 at 16:25

answered Apr 02 '14 at 16:20

AnthonyLambert

8,768
4
37
72

Maximum capacity of Collection different than expected for x86

3 Answers3