Why does malloc() fail when there is enough memory?

Question

I'm using a server with 128GB memory to do some computation. I need to malloc() a 2D float array of size 56120 * 56120. An example code is as follows:

int main(int argc, char const *argv[])
{
    float *ls;
    int num = 56120,i,j;
    ls = (float *)malloc((num * num)*sizeof(float));
    if(ls == NULL){
        cout << "malloc failed !!!" << endl;
        while(1);
    }
    cout << "malloc succeeded ~~~" << endl;
    return 0;
}

The code compiles successfully but when I run it, it says "malloc failed !!!". As I calculated, it only takes about 11GB of memory to hold the whole array. Before I started the code, I checked the server and there was 110GB of free memory available. Why does the error happen?

I also found that if I reduce num to, say 40000, then the malloc will succeed.

Does this mean that there is a limit on the maximum memory that can be allocated by malloc()?

Moreover, if I change the way of allocation, directly declaring a 2D float array of such size, as follows:

int main(int argc, char const *argv[])
{
    int num = 56120,i,j;
    float ls[3149454400];
    if(ls == NULL){
        cout << "malloc failed !!!" << endl;
        while(1);
    }
    cout << "malloc succeeded ~~~" << endl;
    for(i = num - 10 ; i < num; i ++){
        for( j = num - 10; j < num ; j++){
            ls[i*num + j] = 1;
        }
    }
    for(i = num - 11 ; i < num; i ++){
        for( j = num - 11; j < num ; j++){
            cout << ls[i*num + j] << endl;
        }
    }
    return 0;
}

then I compile and run it. I get a "Segmentation fault".

How can I solve this?

Answer 1

The problem is, that your calculation

(num * num) * sizeof(float)

is done as 32-bit signed integer calculation and the result for num=56120 is

-4582051584

Which is then interpreted for size_t with a very huge value

18446744069127500032

You do not have so much memory ;) This is the reason why malloc() fails.

Cast num to size_t in the calculation of malloc, then it should work as expected.

Answer 2

As other have pointed out, 56120*56120 overflows int math on OP's platform. That is undefined behavior (UB).

malloc(size_t x) takes a size_t argument and the values passed to it is best calculated using at least size_t math. By reversing the multiplication order, this is accomplished. sizeof(float) * num cause num to be widened to at least size_t before the multiplication.

int num = 56120,i,j;
// ls = (float *)malloc((num * num)*sizeof(float));
ls = (float *) malloc(sizeof(float) * num * num);

Even though this prevents UB, This does not prevent overflow as mathematically sizeof(float)*56120*56120 may still exceed SIZE_MAX.

Code could detect potential overflow beforehand.

if (num < 0 || SIZE_MAX/sizeof(float)/num < num) Handle_Error();

No need to cast the result of malloc().
Using the size of the referenced variable is easier to code and maintain than sizing to the type.
When num == 0, malloc(0) == NULL is not necessarily an out-of-memory.
All together:

int num = 56120;
if (num < 0 || ((num > 0) && SIZE_MAX/(sizeof *ls)/num < num)) {
  Handle_Error();
}
ls = malloc(sizeof *ls * num * num);
if (ls == NULL && num != 0) {
  Handle_OOM();
}

Answer 3

int num = 56120,i,j;
ls = (float *)malloc((num * num)*sizeof(float));

num * num is 56120*56120 which is 3149454400 which overflows a signed int which causes undefined behavoir.

The reason 40000 works is that 40000*40000 is representable as an int.

Change the type of num to long long (or even unsigned int)

Answer 4

This is in contrast to what others have written, but for me, changing the variable num to size_t from int allows allocation. It could be that num*num overflows the int for malloc. Doing malloc with 56120 * 56120 instead of num*num should throw an overflow error.

Answer 5

float ls[3149454400]; is an array with automatic storage type, which is usually allocated on the process stack. A process stack is limited by default by a value much smaller than 12GB you are attempting to push there. So the segmentation fault you are observing is caused by the stack overflow, rather than by the malloc.