2

I need to allocate a fairly large chunk (or chunks) of memory - several gigabytes. But if I try to allocate a float array of more than 532000000 elements(~2 Gb), I get a runtime error:

terminate called after throwing an instance of 'std::bad_alloc' what(): std::bad_alloc

This is ok:

float* d = new float[532000000];

But this is bad (bad_alloc exception):

float* d = new float[533000000];

Then I tried to allocate another array in addition to the first. It was found that the maximum size of second float array is 195000000 elements (~748 Mb).

This is ok:

float* d = new float[532000000];
float* e = new float[196000000];

This is bad:

float* d = new float[532000000];
float* e = new float[197000000];

I would like to know what are the limitations to the allocated memory in an application and how to avoid them? How can use virtual memory?

My system - 32-bit ubuntu 12.10, compiler - gcc 4.7, RAM - 8GB (~6.5 Gb free)

trincot
  • 317,000
  • 35
  • 244
  • 286
gorill
  • 1,623
  • 3
  • 20
  • 29

1 Answers1

5

You hit the limit of the virtual address space; even if you do have enough physical RAM (that the OS probably can access via PAE, using 36-bit pointers), on a 32 bit system each process still has a 32 bit virtual address space, which means that each process can't map in memory more than 4 GB of memory.

Since usually the upper half of the virtual address space (or the upper 1 GB, it depends from kernel settings) is reserved for the kernel, you will typically have the allocation limit set to ~2 GB, and virtual address space fragmentation can lower this number.

There are various workarounds (for example, on Windows you can use memory-mapped files larger than 4 GB, mapping only a portion of them at a time; probably on Linux you can do the same), but currently the simplest solution is just to move to a 64 bit OS and recompile the application for 64 bit.

Matteo Italia
  • 123,740
  • 17
  • 206
  • 299
  • *move to a 64 bit OS* Does this work if the process is still 32 bit? – ta.speot.is Sep 28 '13 at 23:47
  • @ta.speot.is: it can give it a bit more breath space (the OS stuff gets moved to the upper half 64 bit virtual address space, so the user process can take the full 32 bit virtual address space), but to actually benefit of the 64 bit address space you have to recompile the program for 64 bit. – Matteo Italia Sep 28 '13 at 23:49
  • Is there any limits on 64-bit systems? Can I use all available physical memory? – gorill Sep 28 '13 at 23:51
  • @gorill: a runaway process managed to consume almost all the physical memory and swap space on my Linux machine before getting killed by the OOM killer, so I suppose so. – Matteo Italia Sep 28 '13 at 23:52