How to allocate a single array that does not fit in RAM

Question

I thought I'd never say this. I would like to make my machine run super-slow.

Here's the manner in which I'd like to do this: I'd like to allocate a single large array in F# (Array.init). It should be so large that random accesses into the array should generate page faults. I have 4GB RAM, and running in 64-bit mode. When I allocate 2^29 4-byte integers, runtime throws an out of memory exception.

> Array.init (1 <<< 28) (fun i->i);;
val it : int [] =
  [|0; 1; 2; 3; 4; 5; 6; 7; 8; 9; 10; 11; 12; 13; 14; 15; 16; 17; 18; 19; 20;
    21; 22; 23; 24; 25; 26; 27; 28; 29; 30; 31; 32; 33; 34; 35; 36; 37; 38; 39;
    40; 41; 42; 43; 44; 45; 46; 47; 48; 49; 50; 51; 52; 53; 54; 55; 56; 57; 58;
    59; 60; 61; 62; 63; 64; 65; 66; 67; 68; 69; 70; 71; 72; 73; 74; 75; 76; 77;
    78; 79; 80; 81; 82; 83; 84; 85; 86; 87; 88; 89; 90; 91; 92; 93; 94; 95; 96;
    97; 98; 99; ...|]
> Array.init (1 <<< 29) (fun i->i);;
System.OutOfMemoryException: Exception of type 'System.OutOfMemoryException' was thrown.
   at <StartupCode$FSI_0003>.$FSI_0003.main@()
Stopped due to error
>

For what purpose - any sensible OS will probably page these arrays out to disk pretty quickly. If you really want to do this it is much simpler in C or to just run 100 F# processes which each use 1GB of ram — John Palmer, Feb 21 '12 at 02:42
It's academia, so I am flexing my F# muscle. I would like to avoid C where it can be avoided. — GregC, Feb 21 '12 at 02:44
@JohnPalmer Are you saying there's something as direct as large array alloc that I can do in C but cannot perform at all under CLR? — GregC, Feb 21 '12 at 02:47
as LukeH pointed out, you can't create objects > 2GB in a single process. In C you could just do a giant malloc. The F# solution would be to have lots of programs generating arrays of size `1<<<28` that were started with `System.Diagnostics.Process.Start()` so that they got a different heap — John Palmer, Feb 21 '12 at 02:49

LukeH · Accepted Answer · 2012-02-21T03:00:18.743

The maximum size allowed by the Microsoft CLR for a single object is 2GB, regardless of whether you're running on a 32-bit or 64-bit platform.

You're running up against this hard limit when you try to allocate the array of 2**29 integers. (The array's data would be exactly 2GB, but objects also need a few extra bytes for housekeeping etc, pushing you over 2GB.)

Try allocating a slightly smaller array to allow for those extra few bytes of overhead. I can't remember exactly how much smaller it'd need to be -- a few experiments with (2**29)-3 integers, (2**29)-4 integers, (2**29)-5 integers etc should tell you pretty quickly.

http://stackoverflow.com/questions/1087982/single-objects-still-limited-to-2-gb-in-size-in-clr-4-0 — GregC, Feb 21 '12 at 02:51

How to allocate a single array that does not fit in RAM

1 Answers1