How to allocate memory for a very large two-dimensional array in C

Question

I have a file which has 5 million rows and 4 columns. As usual, I try to read the data in a 5million by 4 array.

long M = 5000000;
double *coordinates[M];
for (i = 0; i < M; i++){
    coordinates[i] = (double *) calloc(3, sizeof(double));
}

But when I run this code, it has segment fault. After searching the answer online, I know it's because the stack doesn't have so much memory. Someone suggested to allocate memory on the heap with malloc if the array is one-dimensional. But I need a two-dimensional array and I really need so much memory, I hope somebody can help me out of this. Thanks a lot.

score 3 · Answer 1 · edited Apr 04 '13 at 15:32

3

Do you really need the extra layer of pointers?

size_t M = 4321098;
double (*coordinates)[4] = calloc(M, sizeof *coordinates);

and you're done. coordinates[234329][3]=3.1415926535; everything works.

edited Apr 04 '13 at 15:32

Fred Foo

355,277
75
744
836

answered Apr 04 '13 at 14:42

jthill

55,082
5
77
137

Your `calloc()` call is incorrect. Otherwise, I like this answer. – Fred Larson Apr 04 '13 at 15:24
@FredLarson: I fixed the `calloc` for jthill, consider an upvote (I just gave one). – Fred Foo Apr 04 '13 at 15:33
@larsmans: Thanks. I guess I could've corrected it too. Upvoted. – Fred Larson Apr 04 '13 at 15:43

score 2 · Accepted Answer · answered Apr 04 '13 at 14:23

2

You could change

double *coordinates[M];

to

double **coordinates = malloc(M * sizeof(*coordinates));

Make sure to free this memory later in the program.

for (i = 0; i < M; i++){
    free(coordinates[i]);
}
free(coordinates);

answered Apr 04 '13 at 14:23

simonc

41,632
12
85
103

Thanks. It works well. BTW, since malloc can't initiate the buffer, is there any other choices that can do allocation and initiation at the same time? – Jin Yan Apr 04 '13 at 14:31
Try calloc! or malloc & then memcpy – Marco Apr 04 '13 at 14:33
1

@JinYan You could use `calloc(M, sizeof(*coordinates))` to allocate memory and initialise it to zero – simonc Apr 04 '13 at 14:33

score 2 · Answer 3 · edited May 23 '17 at 12:13

As everyone said, the problem is that you are trying to allocate that array on the stack of your function.. But, what if you have that array defined as static in your code? (Like a global variable or a static variable in your function?) It doesn't have to be allocated each time your function is called.

That would avoid the calls to malloc and calloc (at the cost of being a fixed size array instead of a dynamic one).

#include <stdio.h>
#include <stdlib.h>
#define ROWS 10000000
#define COLS 4
double coordinates[ROWS][COLS];

int main() {

    for (int i = 0; i < ROWS; i++) {
        for (int j = 0; j < COLS; j++) {
            coordinates[i][j] = 1.0;
        }
    }

    printf("%.4f\n", coordinates[0][2]);

    return 0;
}

The difference between automatic and static variables is that automatic variables live inside your functions (probably they are allocated each time you call your function, whilst an static variable holds its value forever, and probably is allocated only once).

Something like that. I'm using C99, so you can compile it with.. c99

or gcc -std=c99

or CC=c99 make file

Also, in Linux (and I think in windows too) you can increase the stack size of your program.

score 0 · Answer 4 · answered Apr 04 '13 at 14:23

0

You are trying to declare that array on the stack instead of the heap which is leading to a stack overflow error.

Try the following...

double** coordinates = malloc( sizeof( double* ) * M );

answered Apr 04 '13 at 14:23

K Scott Piel

4,320
14
19

How to allocate memory for a very large two-dimensional array in C

4 Answers4