Out of memory when create big array [,]

Question

I have out of memory when I want create big array for example.

 var size = 20000;
 var matrix = new decimal?[size, size];

I have set 64 bit project and uncheck "Prefer 32 bit".

Screen1

I added in App.config

<runtime>
    <gcAllowVeryLargeObjects enabled="true"/>
</runtime>

Screen2

I have 16 GB RAM. What did I forget about?

Possible duplicate of [Create big Two-Dimensional Array](https://stackoverflow.com/questions/9571290/create-big-two-dimensional-array) — 41686d6564 stands w. Palestine, Jan 12 '19 at 21:58
How much memory do you have available? Note that an array of 20.000 * 20.000 **nullable** `Decimal` values would require around 9.6GB of free, continuous, memory to be available to your program. — Lasse V. Karlsen, Jan 12 '19 at 21:59
Note that the size of bytes of `Decimal` is 16. So you are trying to allocate an array that would require 16*20000*20000 bytes (roughly 6 GB). Note that you could get that exception even if you have enough free RAM available. The required 6 GB would need to be available as a continuous memory block within the process address space of your program; if that is not the case the same exception occurs... — , Jan 12 '19 at 21:59
(I just edited my last comment and added more information to it. Please read it again) — , Jan 12 '19 at 22:02
@Lucyper Note the important bit in the comments above is that the memory needs to be *contiguous*. — DavidG, Jan 12 '19 at 22:03
@Lucyper _"I have 16 GB ram"_ That's not an excuse! Even if it did work, that doesn't mean you should do it unless you _**absolutely**_ have to. I'd say you need to rethink your approach. — 41686d6564 stands w. Palestine, Jan 12 '19 at 22:03
As an alternative you can use a dictionary with x,y as key and decimal as value — Sir Rufo, Jan 12 '19 at 22:11
This is not an *actual* out of memory condition though, you have hit upon a hard limit somewhere unrelated to available memory, as is evident by the error message the exception contains, "Array dimensions exceeded supported range". — Lasse V. Karlsen, Jan 12 '19 at 22:14
Side note: When checking for free (uncommitted) RAM in the task manager, don't look at the available RAM, but rather look at the committed memory (from which you can derived the uncommitted memory) to see how much memory at most can really still be allocated/committed. (For example, if the Task Manager shows 9 GB as available RAM, but the amount of uncommitted memory is for example only 7 GB, then no can do, not enough uncommitted memory to allocate from...) — , Jan 12 '19 at 22:15

score 7 · Accepted Answer · edited Jan 12 '19 at 22:40

7

10Gb in one object (24 bytes (size of nullable decimal) * 20000 * 20000) exceeds the default size of a single object in .NET (i.e., the CLR) - see this MSDN documentation "gcAllowVeryLargeObjects Element" on how to exceed that by changing a configuration variable. That article also lists other limits on arrays that you may be interested in, e.g., you can only have 2^32 elements in an array - in case 400M elements isn't enough and you're thinking of taking it to the next level.

A better solution would be a "jagged" array, notated [][] in C# - each row will be a separate object (a 1-D array []) and thus you won't hit any of these limits.

You might want to consider other options. E.g., perhaps your array is sparse? Perhaps it is a matrix with usable attributes e.g., upper triangular? Perhaps you can rethink what you're doing so you don't need a 10Gb array of anything - instead preferring another data structure or even better, if that amount of data really needs to be processed, an external algorithm that stores the data on disk and accesses it in a predictable (e.g., sequential, or multiple-sequential) way?

And finally, I question why you need a giant array of nullable decimal: sounds like you really have a sparse array and should be looking at solutions for sparse arrays (plenty on StackOverflow and Google about those).

edited Jan 12 '19 at 22:40

Wai Ha Lee

8,598
83
57
92

answered Jan 12 '19 at 22:02

davidbak

5,775
3
34
50

1

The size of a *nullable* decimal is 24 bytes though. – Lasse V. Karlsen Jan 12 '19 at 22:03
I need decimal to me problem. But for test i set int?[20000,20000] and I still have the same problem - out of memory – Lucyper Jan 12 '19 at 22:23
@Lucyper - `int?` takes 8 bytes so that is a 3Gb array that still exceeds 2Gb. Maybe start with a smaller problem? Another solution is to use a two dimensional "jagged" array `[][]` where each row will be a separate object. (Instead of being a single object.) It is probably a better solution in any case. Try that, if you insist on going down this route. – davidbak Jan 12 '19 at 22:28
@davidbak I created an algorithm for an array [,]. The array [] [] may be an emergency solution that requires a lot of code changes from my site. – Lucyper Jan 12 '19 at 22:35
1

@Lucyper, you could perhaps create a class featuring an `[int, int]` indexer (https://learn.microsoft.com/en-us/dotnet/csharp/programming-guide/indexers/) as a replacement for your `[ , ]` array, which could perhaps avoid too many code changes. The class would internally manage a collection of smaller arrays (or jagged array, as suggested), with the indexer implementation managing the access to the elements within the individual arrays "hidden" inside this class. – Jan 12 '19 at 22:49
@davidbak Finally, I change the code to [] []. Thank you for your help – Lucyper Jan 12 '19 at 23:05

Out of memory when create big array [,]

1 Answers1