3

Even though this post says it should work, if you create an int array of size Int32.MaxValue, it throws an OutOfMemoryException: Array dimensions exceeded supported range.

From my testing, it seems like the maximum size that an array can be initialized to is Int32.MaxValue - 1048576 (2,146,435,071). 1048576 is 2^20. So only this works:

var maxSizeOfIntArray = Int32.MaxValue - 1048576;
var array = new int[maxSizeOfIntArray];

Does any one know why? Is there a way to create a larger integer array?

PS: I need to use arrays instead of lists because of a Math.Net library that only returns arrays for sets of random numbers that are cryptographically secure pseudo random number generator

Yes I have looked at the other questions linked but they are not correct as those questions say the largest size is Int32.MaxValue which is not the same as what my computer lets me do

Yes, I do know the size of the array will be 8GB, I need to generate a data set of billions of rows in order to test the randomness with the die harder suite of tests

I also tried the option of creating a BigArray<T> but that doesn't seem to be supported in C# anymore. I found one implementation of it, but that throws an IndexOutOfRangeException at index 524287, even though I set the array size to 3 million.

Saamer
  • 4,687
  • 1
  • 13
  • 55
  • Times 4 bytes per int is quite a chunk of memory. Are you *really* sure you need that much? – Hans Kesting Apr 21 '21 at 19:38
  • 3
    _a library that only returns arrays_ then why would __you__ need to create the array? – TaW Apr 21 '21 at 19:39
  • https://stackoverflow.com/questions/30895549/cant-create-huge-arrays – Dmitry Bychenko Apr 21 '21 at 19:50
  • As @BrianWilson explains in his answer, that array would consume 8GB of memory. That's a lot. What is your use case for an array of such an immense dimension. You may be better off with a _sparse array_ implementation (the easiest of which is implemented over top of a `Dictionary` for a sparse array of type `T`). I suspect you cannot allocate more than the GC's _ephemeral segment size_ in a single allocation: https://learn.microsoft.com/en-us/dotnet/standard/garbage-collection/fundamentals#ephemeral-generations-and-segments – Flydog57 Apr 21 '21 at 20:34
  • @HansKesting I need it to create a super big file of >10 billion numbers, in order to test the bias in a random number generator using `dieharder` – Saamer Sep 26 '21 at 01:47
  • When you need to create a *file*, then you could just write to the file without containing everything in memory first – Hans Kesting Sep 26 '21 at 06:16

1 Answers1

-4

An Int32 is 32 bits, or 4 bytes. The max value of an Int32 is 2,147,483,647. So, if you could create an array of 2,147,483,647 elements, where each element is 4 bytes, you would need a contiguous piece of memory that is 8GB in size. That is ridiculously huge, and even if your machine had 128GB of RAM (and you were running in a 64-bit process), that would be outside of realistic proportions. If you really need to use that much memory (and your system has it), I would recommend going to native code (i.e., C++).

Brian Wilson
  • 86
  • 3
  • 6
    How is using "native" code supposed to help memory consumption for the storage of integers? – Mark Benningfield Apr 21 '21 at 19:42
  • 4
    How does this answer the OP's question? And how would using Native code change anything? – Johan Donne Apr 21 '21 at 19:42
  • Mark Benningfield, I'm sorry, if you want an explanation of how/why native code gives you more power and flexibility on memory usage and handling, please ask a question. I'll say this, though---.NET (and .NET core) are virtual machines that are written themselves in native code. You can do anything .NET can do in native code, and more. You are much closer to the hardware. – Brian Wilson Apr 21 '21 at 19:55
  • Johan Donne: same thing: I'm not going to use a StackOverflow comment to explain how native code works. Feel free to ask a question. – Brian Wilson Apr 21 '21 at 19:56