Why do very large stack allocations fail despite unlimited ulimit?

Question

The following static allocation gives segmentation fault

double U[100][2048][2048];

But the following dynamic allocation goes fine

double ***U = (double ***)malloc(100 * sizeof(double **));

for(i=0;i<100;i++)
{
    U[i] = (double **)malloc(2048 * sizeof(double *));
    for(j=0;j<2048;j++)
    {
        U[i][j] = (double *)malloc(2048*sizeof(double));
    }
}

The ulimit is set to unlimited in linux.

Can anyone give me some hint on whats happening?

Are you absolutely sure that the stack size is unlimited? Also, have you enough memory? The `malloc`ed memory may not all be backed by physical memory. — Daniel Fischer, Mar 16 '13 at 11:02
yes I had set the ulimit to unlimited. Is there any chance that operating systems may enforce a particular limit? — Anup Buchke, Mar 16 '13 at 11:05
You're asking for 3 355 443 200 bytes of memory on the stack. Is there enough memory available in your system? — Angew is no longer proud of SO, Mar 16 '13 at 11:08
yes...I am using the array in the code following the declaration. Each and every location is accessible — Anup Buchke, Mar 16 '13 at 11:10
@Angew You may be right?...But how does it differentiate betwn static and dynamic..how can i find that out...if that is the case i can have my code check the system available memory and decide the allocation method — Anup Buchke, Mar 16 '13 at 11:13
Local variables will be allocated on the stack (limited in size). Malloc will return a address to anywhere in the memory (with enough space). — flyingOwl, Mar 16 '13 at 11:16
Dynamically allocated multidimensional arrays don't need to be contiguous in memory, statically allocated multidimensional arrays do. — JSQuareD, Mar 16 '13 at 11:19
Can malloc allocate space for 100*2048*2048 doubles or does it just return NULL? The issue might be more related to memory (address space) fragmentation and less related to static vs dynamic. — Andrei, Mar 16 '13 at 11:30
Probably duplicate of http://stackoverflow.com/q/15165635/841108 and/or http://stackoverflow.com/q/15165635/841108 — Basile Starynkevitch, Mar 16 '13 at 11:55
You don't need to cast the return value of `malloc()` in a C program. — Carl Norum, Mar 16 '13 at 16:07
`malloc`ing large memory blocks usually works (on Linux) because the kernel doesn't actually ensure that every locations is available. If the kernel has no more memory available, the infamous OOM killer blasts some applications to get some memory. https://stackoverflow.com/a/19991682/854672 — MauganRa, Jan 19 '14 at 14:00

teppic · Accepted Answer · 2013-03-16T15:32:02.007

When you say the ulimit is set to unlimited, are you using the -s option? As otherwise this doesn't change the stack limit, only the file size limit.

There appear to be stack limits regardless, though. I can allocate:

double *u = malloc(200*2048*2048*(sizeof(double)));  // 6gb contiguous memory

And running the binary I get:

VmData:  6553660 kB

However, if I allocate on the stack, it's:

double u[200][2048][2048];

VmStk:   2359308 kB

Which is clearly not correct (suggesting overflow). With the original allocations, the two give the same results:

Array:  VmStk:   3276820 kB
malloc: VmData:  3276860 kB

However, running the stack version, I cannot generate a segfault no matter what the size of the array -- even if it's more than the total memory actually on the system, if -s unlimited is set.

EDIT:

I did a test with malloc in a loop until it failed:

VmData: 137435723384 kB  // my system doesn't quite have 131068gb RAM

Stack usage never gets above 4gb, however.

score 4 · Answer 2 · answered Mar 16 '13 at 11:49

4

Assuming your machine actually has enough free memory to allocate 3.125 GiB of data, the difference most likely lies in the fact that the static allocation needs all of this memory to be contiguous (it's actually a 3-dimensional array), while the dynamic allocation only needs contiguous blocks of about 2048*8 = 16 KiB (it's an array of pointers to arrays of pointers to quite small actual arrays).

It is also possible that your operating system uses swap files for heap memory when it runs out, but not for stack memory.

answered Mar 16 '13 at 11:49

JSQuareD

4,641
2
18
27

The malloced space also needs to continuous. It's by definition – stdcall Mar 16 '13 at 11:55
4

@Mellowcandle, Yeah, but with the dynamic allocation, there have to be 204800 contiguous 2048-byte areas (not counting the higher pointer levels because they are small in comparison), whereas with the static allocation ALL of the memory would have to be contiguous. – us2012 Mar 16 '13 at 11:58
@Mellowcandle Yeah, but each call to malloc only allocates a small amount of memory. – JSQuareD Mar 16 '13 at 11:58
The stack memory only needs to be continous in *virtual* memory address space and kernel can create virtual memory by re-organizing physical memory as it sees fit. However, if the memory is highly fragmented that re-organizing needs a lot of "page tables" (the actual virtual memory implementation of most CPUs) and system may easily fail to allocate both the 3.125 GiB of data plus the required page tables. For highly fragmented memory the page tables alone can take gigabytes of RAM. And if the system had 32 bit kernel, the *virtual address space* may fill up, too, even if system had free RAM! – Mikko Rantalainen Aug 23 '21 at 07:38

score 2 · Answer 3 · edited May 23 '17 at 11:50

There is a very good discussion of Linux memory management - and specifically the stack - here: 9.7 Stack overflow, it is worth the read.

You can use this command to find out what your current stack soft limit is

ulimit -s

On Mac OS X the hard limit is 64MB, see How to change the stack size using ulimit or per process on Mac OS X for a C or Ruby program?

You can modify the stack limit at run-time from your program, see Change stack size for a C++ application in Linux during compilation with GNU compiler

I combined your code with the sample there, here's a working program

#include <stdio.h>
#include <sys/resource.h>

unsigned myrand() {
    static unsigned x = 1;
    return (x = x * 1664525 + 1013904223);
}

void increase_stack( rlim_t stack_size )
{
    rlim_t MIN_STACK = 1024 * 1024;

    stack_size += MIN_STACK;

    struct rlimit rl;
    int result;

    result = getrlimit(RLIMIT_STACK, &rl);
    if (result == 0)
    {
       if (rl.rlim_cur < stack_size)
       {
           rl.rlim_cur = stack_size;
           result = setrlimit(RLIMIT_STACK, &rl);
           if (result != 0)
           {
              fprintf(stderr, "setrlimit returned result = %d\n", result);
           }
       }
   }    
}

void my_func() {
   double U[100][2048][2048];
   int i,j,k;
   for(i=0;i<100;++i)
    for(j=0;j<2048;++j)
        for(k=0;k<2048;++k)
            U[i][j][k] = myrand();
   double sum = 0;
   int n;
   for(n=0;n<1000;++n)
       sum += U[myrand()%100][myrand()%2048][myrand()%2048];
   printf("sum=%g\n",sum);
}

int main() {
   increase_stack( sizeof(double) * 100 * 2048 * 2048 );

   my_func();

   return 0;
}

If the stack size has already been set to unlimited (as the question says), this would actually reduce it to 100*2048*2048*sizeof(double). — teppic, Mar 16 '13 at 16:53
The code checks to see if the requested limit is less than the existing limit and it doesn't decrease it in that case. So if the stack limit is in fact unlimited, this code will not decrease it. — amdn, Mar 16 '13 at 17:24
Oh yes, I didn't see that part. This code is only useful though if the user hasn't increased the soft stack limit. — teppic, Mar 16 '13 at 17:28

score -3 · Answer 4 · edited May 23 '17 at 11:57

-3

You are hitting a limit of the stack. By default on Windows, the stack is 1M but can grow more if there is enough memory.

On many *nix systems default stack size is 512K.

You are trying to allocate 2048 * 2048 * 100 * 8 bytes, which is over 2^25 (over 2G for stack). If you have a lot of virtual memory available and still want to allocate this on stack, use a different stack limit while linking the application.

Linux: How to increase the gcc executable stack size? Change stack size for a C++ application in Linux during compilation with GNU compiler

Windows: http://msdn.microsoft.com/en-us/library/tdkhxaks%28v=vs.110%29.aspx

edited May 23 '17 at 11:57

Community

1
1

answered Mar 16 '13 at 11:53

Valeri Atamaniouk

5,125
2
16
18

2

-1 : The question explicitly says that `ulimit` stack size is unlimited. – us2012 Mar 16 '13 at 11:56
@us2012 And what about stack size specified at executable assembly? I mean linker flags? – Valeri Atamaniouk Mar 16 '13 at 12:13

Why do very large stack allocations fail despite unlimited ulimit?

4 Answers4