0

I'm writing a program in C on windows that launches 30 threads, each of which needs an array of int16_t.

The size is calculated before the thread function is called and in the example I'm working with it's around 250 millions. This is around 15GB, which should not be a problem, because I have 128GB ram available.

I've tried using both malloc and calloc inside the thread function, but over half of the allocations return NULL with errno set to 12 (enomem).

With a small number of threads (up to 3) it works fine though, same if I just use 1 thread and allocating an unreasonably big array.

My next attempt to solve this issue was to create an array of pointers in the main, allocate the arrays there and pass them as argument to the thread, same thing happened.

So from these results my best guess would be it can't allocate contiguous blocks of memory of that size, so I also tried allocating many smaller arrays, which obviously didn't work either. Is this an expected behaviour or am I doing something wrong?

Déjà vu
  • 28,223
  • 6
  • 72
  • 100
incantation
  • 13
  • 1
  • 6
  • 2 * 250 * 10^6 * 30 = 15*10^9 bytes. Sanity check your spec before you start coding away maybe. – Lundin Nov 22 '18 at 15:33
  • guess I should've mentioned I have 128 gb of ram available – incantation Nov 22 '18 at 15:37
  • 2
    I guess what you are actually asking is this: https://stackoverflow.com/questions/11891593/the-maximum-amount-of-memory-any-single-process-on-windows-can-address – Lundin Nov 22 '18 at 15:42
  • Also a [mcve] might help. Just the part that does the failing creation of the array of pointers in `main`. – Jabberwocky Nov 22 '18 at 15:44
  • 1
    Are you compiling as 32 bit or 64 bit code? A 32 bit process cannot address more than 4 GB of memory, even if your computer has 128Gb or ram. – Jabberwocky Nov 22 '18 at 15:48
  • @Jabberwocky i always wondered when i cannot have more than 4 GB of ram for my program, how is it possible for some programs e.g. Games, to get more RAM? My games usually take up to 7-8 GB. Is it because these processes launch other processes which run in their context and windows only tells the sum of the memory consumption? – Yastanub Nov 22 '18 at 15:51
  • @Yastanub probably because they are 64 bit programs. Are you using Visual Studio? – Jabberwocky Nov 22 '18 at 15:52
  • @Jabberwocky i just noticed i read the posted link from Lundin with a little bias. I read it again and now it is clear. I remember now that i actually once found out i was compiling 32-bit applications and thats why my experiments always failed. – Yastanub Nov 22 '18 at 15:56
  • @Jabberwocky I was in fact compiling with the 32 bit compiler, tried it with the 64 bit version and it seems to work fine now, my understanding was that it depended on the system and not on the program itself, which apparently isn't the case. Thank you very much for your help! – incantation Nov 22 '18 at 16:20

0 Answers0