Large 2D array in C - Stack OverFlow error

Question

I haven't used C in long time, and I'm having an issue filling a 2D array from a CSV. The file format is like this:

Node, In, Out

1,200,10393

...

This is essentially an array representation of a linked list. There are 150000 elements and whenever I try to fill the array I get an error saying "Unhandled exception at 0x000000013facb957 in main.exe: 0xC00000FD: Stack overflow." I'm on a 64-bit machine with 16GB of RAM and I'm using VS C++ 2010 Express with an x64 build configuration.

int main(int argc, char *argv[])
{

int counter = 0;
char line [ 1024 ];
int map[150000][2] = {0};
char *comma = ",";
char *token;
int index;
int in, out;
char* end;
int nodeID;
FILE *fp;

fp = fopen("mapsorted.txt","r"); // read mode

if( fp == NULL )
{
  perror("Error while opening the file.\n");
  exit(EXIT_FAILURE);
}

//Skip header line
fgets ( line, sizeof line, fp );

while ( fgets ( line, sizeof line, fp ) != NULL) /* read a line */
{
    //first part - the index for storage
    token = strtok(line,comma);
    index = strtol(token,&end,10);

    //second part
    token = strtok(NULL,comma);
    in = atoi(token);

    //third part
    token = strtok(NULL,comma);
    out = atoi(token);

    //store in array
    map[index][0] = in;
    map[index][1] = out;
}
fclose ( fp );
}

The code seems to work when I allocate a smaller array, but fails when it is this large. I think I should have enough memory to be able to handle an array of this size.

I thought Windows had increased the stack size from the ridiculously small 1MB default. — Daniel Fischer, Dec 30 '12 at 18:29
Does this answer your question? [Getting a stack overflow exception when declaring a large array](https://stackoverflow.com/questions/571945/getting-a-stack-overflow-exception-when-declaring-a-large-array) — Nate Eldredge, Feb 09 '21 at 22:39

score 7 · Accepted Answer · 2012-12-30T18:37:38.343

7

int map[150000][2];

seems at least 2 * 4 * 150000 bytes (assuming a modern 32-bit architecture), which is roughly 1.2MB. Knowing that modern OSes typically set up a stack size of a few megabytes, this could actually be the problem. Your computer having several gigabytes of RAM doesn't mean all of it can be consumed by your process, especially not on the stack. For large arrays, try malloc()ating some memory on the heap:

int (*map)[2] = malloc(sizeof(*map) * 150000);

or

int *map = malloc(150000 * 2 * sizeof(*map));

(pay attention to the dimensions in the second case!), or declare it as static to move it out of stack space:

static int map[150000][2];

Or simply make it a global variable to achieve a similar behavior:

int map[150000][2];

void foo()
{
    ...
}

edited Dec 30 '12 at 18:37

answered Dec 30 '12 at 18:29

Or even just let the compiler crunch the math for you: `int (*ar)[2] = malloc(sizeof(int[150000][2]));` =P – WhozCraig Dec 30 '12 at 18:59
@WhozCraig Not preferred, risky. The less risk is in the way I used. – Dec 30 '12 at 19:01
Genuinely curious. whats the risk? (i.e. should I start perusing code I've done this in and chain-retool)? – WhozCraig Dec 30 '12 at 19:02
@WhozCraig When the dimensions of the array and/or the base type changes, and one forgets to (don't say you don't forget it, we all do) change that in the malloc part as well, you get into trouble. And well, you don't have to go back and rewrite all code, but you can get used to this from now on, if you wish. – Dec 30 '12 at 19:04
I was about to post before your response to nm. My alloc is type-based -only, not var-type based like yours. Touche, and i concur with you this regard. btw. The behavior is fine so long as you don't *change* it, whereby you then incur risk, and I agree. (and +1, btw). – WhozCraig Dec 30 '12 at 19:06
Thanks. I knew I was doing something stupid, because I came up with the same size for the array of ~ 1.2MB. – user1489497 Dec 30 '12 at 19:24

score 2 · Answer 2 · answered Dec 30 '12 at 18:28

2

The array is too big to fit in the stack. Try moving it outside of the function:

static int map[150000][2] = {0};
int main(int argc, char *argv[])
{

and so on.

answered Dec 30 '12 at 18:28

Erik Ekman

2,051
12
13

...or to allocate it dynamically on the free store – K-ballo Dec 30 '12 at 18:28
It's hardly a big change, the rest of the code that deals with `map` won't change... – K-ballo Dec 30 '12 at 18:30
@K-ballo Well, it would, since 2D access syntax would have to be changed. – Dec 30 '12 at 18:31
@H2CO3: How would it change?? – K-ballo Dec 30 '12 at 18:32
@K-ballo You can't subscript a single pointer twice, `int *map` and `map[index][0]` don't fit together. – Dec 30 '12 at 18:33
Exactly. Wrap the two fields in a `struct` or something and then `malloc` will do fine. – Erik Ekman Dec 30 '12 at 18:34
1

@H2CO3 `int (*map)[2] = malloc(150000*sizeof *map);`. – Daniel Fischer Dec 30 '12 at 18:34
@H2CO3: `static int (*map)[2] = (int (*)[2])malloc( sizeof(int[2]) * 15000 );`? – K-ballo Dec 30 '12 at 18:34
@K-ballo Well, that's not what I wrote, if I recall correctly. (And you don't need the cast.) – Dec 30 '12 at 18:35
@H2CO3: That's how you write it to be dynamically allocated, and still use it the same everywhere... – K-ballo Dec 30 '12 at 18:36
@K-ballo Thanks, I just realized. – Dec 30 '12 at 18:36
If someone says they havent used C in a long time I think they prefer something simpler.. – Erik Ekman Dec 30 '12 at 18:36

Large 2D array in C - Stack OverFlow error

2 Answers2