Is there any limits for matrix allocation using malloc in C?

Question

I am using dynamical allocation because I need some memory space to define 3 matrices. When I use 2 matrices (please see the minimal working environment below), i.e. I put the third matrix as a comment, the program work very well and gives the desired results. However, when I include the third matrix, I get a segmentation fault, core dumped! I don't know if there are limits in the dynamical allocation using malloc. Any help would be highly appreciated. Thank you!

#include<stdio.h>
#include<malloc.h>

int main(int argc, char **argv)
{
    int rows=160, cols=1999000;

    float *mat1 = (float *)malloc(rows * cols * sizeof(float));
    float *mat2 = (float *)malloc(rows * cols * sizeof(float));
    //float *mat3 = (float *)malloc(rows * cols * sizeof(float));

    int i, j;

    for(i=0;i<rows;i++) 
    {
        for(j=0;j<cols;j++) 
        {
            mat1[i * cols + j]=i+j;
        mat2[i * cols + j]=i-j;
        //mat3[i * cols + j]=i*j;
        }
    }

    for(i=0;i<2;i++) 
    {
        for(j=0;j<6;j++) 
        {
            printf("R[%d][%d] = %f\n", i, j, mat2[i * cols + j]); 
        }
    } 
}

The natural limit is of course the amount of virtual memory your system have. — Some programmer dude, Mar 14 '14 at 08:32
@JoachimPileborg but how do 2 matrices are normally accepted whereas 3 not?? Do you have any numbers of what one could allocate in term of memory? — , Mar 14 '14 at 08:34
Each allocation you make in your example program is well over a GB in size. Making three allocation is over 3.5GB (which is more than is possible to have on a 32-bit Windows machine). — Some programmer dude, Mar 14 '14 at 08:34
Also, the memory for a single allocation have to be contiguous, so even if you're on a 64-bit machine with plenty of virtual memory, there may not be a chunk of virtual memory big enough. — Some programmer dude, Mar 14 '14 at 08:36
maybe you should rethink your design, consider all the effect on other processes in the system. — AndersK, Mar 14 '14 at 08:38
@Claptrap Each process normally has its own virtual address space. So there shouldn't be an effect on other processes (provided that there is enough RAM). — glglgl, Mar 14 '14 at 08:53
BTW, see also [Do I cast the result of malloc?](http://stackoverflow.com/q/605845/296974) — glglgl, Mar 14 '14 at 08:55
Windows uses lazy allocation; it will give you an answer to `malloc` and not actually allocate virtual memory until you start using it; and then kill the process if it fails. Not sure which other OS's do the same. — M.M, Mar 14 '14 at 08:56
@MattMcNabb This counts for physical memory. If the address space is exhausted, it has no other chioce than failing early... — glglgl, Mar 14 '14 at 09:14

n0p · Accepted Answer · 2014-03-14T09:26:19.813

2

Most likely as said in comment you are running out of memory. Check the return of your mallocs: my guess is that your 3rd allocation returns NULL and this triggers the segfault when you try to affect a value to it.

[EDIT]

float *mat3 = (float *)malloc(rows * cols * sizeof(float));
if(mat3 == NULL)
{
    printf("Not enough memory");
    return ENOMEM;
}

[EDIT2]

Also as been said in the comment, using malloc forces you to get contiguous memory. Considering the huge amount of space you need, this design might not be appropriate. You might want to use linked list for example, which allow you to "fragment" your memory.

Anyway, ensure your hardware config (available memory) can support what you code want to do.

edited Mar 14 '14 at 09:26

answered Mar 14 '14 at 08:37

n0p

3,399
2
29
50

1

By checking the return value of malloc. <_ – Devolus Mar 14 '14 at 08:43
I added an example as an EDIT – n0p Mar 14 '14 at 08:44
Yes Coconop. There is not enough memory. Is there any idea to have such a memory? – Mar 14 '14 at 08:56
I think that I would use a variable type other than float! Is it possible to define the variable size (even if it is integer) so that I could use more space momory? Thank you!! best – Mar 14 '14 at 09:10
@Strömungsmechanik Do you definitely need so much memory? If the matrices are as simple as indicated in your question, maybe calculating the data "live" is more appropriate... – glglgl Mar 14 '14 at 09:13
@glglgl I don't know if you mean by "live" static allocation. I tried also to define the variables statically but I failed? – Mar 14 '14 at 09:17
@Strömungsmechanik No. I suppose that afterwards you want to access the matrices. Instead of doing so with `mat1[i * cols + j]`, you could write `i+j` directly where you need it. – glglgl Mar 14 '14 at 09:19
@glglgl In fact,that was just an example. The variable must be stored in order to be used after for certain operations. I am thinking right to use an other kind of variables rather than float which is less memory consuming. Do you have any idea about what one could use? Even not reals would be a nice choice! – Mar 14 '14 at 09:26
I have just checked that even short cannot do the job. No enough memory always!! – Mar 14 '14 at 09:29
@Strömungsmechanik Well if you don't have enough, there is nothing you can do except upgrading your hardware (buying RAM) – n0p Mar 14 '14 at 09:29
@Coconop As already stated, RAM alone won't help here. If it is a 32 bit process, you might be bitten. – glglgl Mar 14 '14 at 09:45
@Strömungsmechanik Do you need all 3 matrices at the same time in memory? If not, you could store them in files and load them on demand. – glglgl Mar 14 '14 at 09:46
@glglgl True, I was not aware of this limitation. – n0p Mar 14 '14 at 09:48

score 2 · Answer 2 · answered Mar 14 '14 at 08:53

First of all you are trying to allocate 1 249 375 bytes (about 1.2 GiB) with each malloc, this can fail even if there is enough free memory because there might be no continuous memory block of this size due to memory fragmentation.

Secondly, if your program is 32-bit then, on Linux, depending on your kernel and settings single process can use up to 3-4 GiB of memory (most likely 3). Since you are allocating more then 1 GiB with each of 3 calls you might exceed this limit. On Windows 32-bit process can use up to 2 GiB by default and up to 3 GiB with 4-Gigabyte Tuning. For 64-bit programs those limits are much larger, but your allocations can still fail because of memory fragmentation.

Bottom line, you should be aware of those limits and check return values of malloc.

@Coconop Thank you for your powerful explanations! I am working on Ubuntu with a 32-bits. It seems that there is not enough memory. Therefore, I may change my thinking strategy! — , Mar 14 '14 at 09:04

Dexters · Answer 3 · 2014-03-14T09:02:43.683

1

You are requesting

7.14898109436 GB of memory for 3 matrices.. Does your computer have so much memory ?

So the memory requirement is very huge for your application and hence your issue with 3 matrices

(Did i get the calculation right?:))

https://www.google.com/search?q=1999000*160*8*3bytes+in+gb

edited Mar 14 '14 at 09:02

answered Mar 14 '14 at 08:49

Dexters

2,419
6
37
57

Is there any limits for matrix allocation using malloc in C?

3 Answers3