1

I'm trying to initialize a jagged array of 120 000 x 4600 for a data mining algorithm but I have an OutOfMemoryException exception

double[][] array = new double[120000][];
for (int i = 0; i < array.Length; i++)
{
    array[i] = new double[4600];
}

it throws when i is around 49 000

I know .Net is limited to 2GB per object, but I thought that the array here would be a list of addresses to an array of double. So it wouldn't be a big single object.

I thought that's why Jon Skeet suggests an jagged array for this question OutOfMemoryException on declaration of Large Array

I don't think that I understand his answer.

Is it one big object and if it's not why does it throw an exception.

Thank you

Community
  • 1
  • 1
Marc
  • 16,170
  • 20
  • 76
  • 119
  • 2
    Did you compile as 64 bits? Also make sure that "Prefer 32-Bit" isn't set in the Build options. – Matthew Watson Nov 04 '13 at 16:30
  • 4,600 * 49,000 (the iteration at which it throws) is 225,400,000 elements. That multiplied by 8 bytes (the size of a `double`) is 1,803,200,000 bytes. And you don't know why it throws `OutOfMemoryException`? – Daniel Daranas Nov 04 '13 at 16:31
  • `4600 * 120000 * 8 bytes = 4.11 GB` – clcto Nov 04 '13 at 16:32
  • yeah, I thought it would be 120000 arrays of 4200, instead of one big 4gb object – Marc Nov 04 '13 at 16:35
  • 1
    To all the people saying he will have a 4GB data structute, that is only true if he declared the array `[,]`, by making a jagged array with `[][]` it is one array of `120000` references and `120000` arrays that are `4600 * 8` bytes large. Marc, you are correct, the commenters are wrong. – Scott Chamberlain Nov 04 '13 at 16:36
  • 1
    @Scott The commenters saying that you have one single 4 GB data structure are wrong, yes - only that there are _no_ such commenters. – Daniel Daranas Nov 04 '13 at 16:38
  • 1
    @DanielDaranas The people I was addressing have deleted their comments. – Scott Chamberlain Nov 04 '13 at 16:41
  • @MatthewWatson Thank you! that was the problem. – Marc Nov 04 '13 at 16:42

1 Answers1

2

If it is 32-bit application, You are rightly getting OutOfMemoryException. For this size requirements you need to target X64.

At i = 49000, Total memory = 49000*4600*8 = 1803200000 bytes = ~1.68GB.

Now For 32-bit applications (targeted X86), Total User Memory avaiable to an application is 2GB (unless the application is Large address aware, .NET application - Large Address Aware and OS is also enabled for this. Ex: (for)Vista. Then there is some CLR overhead, then application overhead.

At i = 120000, You need total memory as Total memory = 120000*4600*8 = 1803200000 bytes = ~4.11GB. (Platform target should be X64)

Tilak
  • 30,108
  • 19
  • 83
  • 131
  • Also, the address space is usually pretty fragmented in 32 bit processes which limits the size of the biggest contiguous block. – Brian Rasmussen Nov 04 '13 at 16:57