2

Is the maximum heap size of a program in C fixed or if I keep malloc-ing it will at some point start to overflow?

Code:

 while(connectionOK) //connectionOK is the connection with server which might be forever
 {
    if(userlookup_IDNotFound(userID))
     user_struct* newuser = malloc(getsize(user_struct));
     setupUserAccount(newuser);
 }

I am using gcc in ubuntu/ linux if that matters. I know something like getrlimit but not sure if it gives heap size. Although it does give the default stack size for one of the options in the input argument. Also valgrind is probably a good tool as suggested here how to get Heap size of a program but I want to dynamically print an error message if there is a heap overflow. My understanding was the process address space being allocated by the OS (which is literally allowed to use the whole memory if it wants to) at the beginning of the process creation but I am not sure if it is dynamically given more physical memory once it requests for additional memory.

Community
  • 1
  • 1
as3rdaccount
  • 3,711
  • 12
  • 42
  • 62

3 Answers3

3

The heap never overflows it just runs out of memory at a certain point (usually when malloc() returns NULL) So to detect out of memory just check the return value of the malloc() call.

if (newuser == NULL)
{
    printf("OOM\n");
    exit(1); /* exit if you want or can't handle being OOM */
}

malloc() internally will request more memory from the OS so it expands dynamically so it's not really fixed size as it will give back pages to the OS that it no longer needs as well as requesting more at any given time that it requires them.

Jesus Ramos
  • 22,940
  • 10
  • 58
  • 88
  • 1
    Alas, some systems (like Linux) will overcommit which means that `malloc` may *not* return `NULL` when it ought to, and instead when you try to use overcommitted memory the OOM killer is invoked. – dmckee --- ex-moderator kitten Feb 04 '13 at 22:55
  • @dmckee The old Linux OOM killer, this is due to the zero page and on demand allocation where pages will fault when first accessed. I'm not sure if there is a way around that exactly (maybe `MAP_POPULATE` flag for `mmap`?) – Jesus Ramos Feb 04 '13 at 22:57
  • You can apparently disable overcommit. Not sure if that is at boot time, or what? Googling for `disable linux overcommit` turns up a variety of links – dmckee --- ex-moderator kitten Feb 04 '13 at 23:01
  • @dmckee I don't recommend doing that unless you want half your programs to stop running. On demand allocation works well for things like java based processes where java decides to allocate a ton of memory at once and never use it. Technically anything that overcommits memory will stop running (almost everything anyway). – Jesus Ramos Feb 04 '13 at 23:03
2

Technically what malloc allocates on most systems is not memory, but address space. On a modern system you can easily allocate several petabytes of address space with malloc and malloc will probably always return a non null pointer. The reason behind this is, that most OS actually perform memory allocation only when a piece of address space is actively modified. As long as it sits there untouched, the OS will just make a note that a certain area of a process address space has been validly reserved for future use.

This kind of bahavior is called "memory overcommitment" and is of importance when maintaining Linux systems. If can happen, that theres more memory allocated than available for some time, and then some program will actually write to some of the overcommited memory. What then happens is, that the so called "Out Of Memory Killer" (OOM killer) will go on a rampage and kills those processes it sees most apropriate for; unfortunately it usually are those processes you don't want to loose under any circumstances. Databases are known to be among the prime targets of the OOM killer.

Because of this, it's strongly recommended to switch of memory overcommitment on high availability Linux boxes. With disabled memory overcommitment disabled, each request for address space must be backed by memory. In that case malloc will actually return 0 if the request can not be fullfilled.

datenwolf
  • 159,371
  • 13
  • 185
  • 298
  • 1
    Problem with that is address space hungry applications (typically java based ones) will tend to eat most of the available memory on the system while not actually using it in some cases. – Jesus Ramos Feb 04 '13 at 23:15
  • @JesusRamos: The reason for that is, that programs that exhibit that behavior will initialize allocated address space with some value. This the OS sees as memory modification and will allocate actual memory for it then. – datenwolf Feb 05 '13 at 01:53
1

at some point, malloc() will return NULL, when system will run out of memory. then when you try to dereference that, your program will abort executing.

See what happens when you do malloc(SIZE_MAX) a few times :-)

Aniket Inge
  • 25,375
  • 5
  • 50
  • 78
  • 1
    SIZE_MAX is almost certainly going to fail even the first time, as that tends to be the the entire adressable range in that processor (mode), which means the heap can't cope with that. – Mats Petersson Feb 04 '13 at 23:02