C When trying to calloc maximum free memory, no NULL return

Question

I'm trying to initialize as much memory as possible (all of free memory), than sleep for 10 seconds and free it up. Calloc initializes it and it goes to a bit over 7800 MB, out of 8GB that I have, so I think it does the job, but the problem begins when (as long as I was reading forums and stuff) the OOM killer comes and kills it. So the process get's killed instead of calloc returning NULL. Is there any fix to that? How to stop it right before killing or how to return NULL when it runs out of memory?

int main() {

int *pointer;
int megabajti=1048576; //MB
int velikost_strani=4096; //page size
long long int i=0;
while(1)
{
    pointer=calloc(velikost_strani,sizeof(int));
    printf("Trenutno alociram %lld MB\n",i*velikost_strani*sizeof(int)/megabajti);
    if(pointer==NULL)
    {
        printf("Max velikost je %lld MB\n",i*velikost_strani*sizeof(int)/megabajti);
        free(pointer);
        sleep(10);
    }
++i;
}
return 0;
}

I just edited the code and used while instead. Thank you, but the result is still the same — S-K, Jun 05 '16 at 13:31
Passing data having wrong type to `printf()` will invoke *undefined behavior*. Casting to desired type will make it safe. — MikeCAT, Jun 05 '16 at 13:32
this is all system dependent stuff here... OOM killer isn't a posix construct exactly... — Grady Player, Jun 05 '16 at 13:33
You *are* using `calloc()` to initialise the memory already. So I deleted my comment. Sry. Any ways, you might like to make yourself comfortable with the concept of "*Overcommitting*". — alk, Jun 05 '16 at 13:33
MikeCAT, Might telling me how to fix this? GradyPlayer, How to fix the problem than? alk, no problem mate ^^ — S-K, Jun 05 '16 at 13:35
@matt If I change calloc to malloc, the memory won't be initialized. Won't be set to 0 I could use memset than, but it still gives me the same result — S-K, Jun 05 '16 at 13:37
You could switch to some OS/compiler dependent stuff, such as "memsize()" or whatever. — Matt, Jun 05 '16 at 13:42
[Please see this discussion on why not to cast the return value of `malloc()` and family in `C`.](http://stackoverflow.com/q/605845/2173917). — Sourav Ghosh, Jun 05 '16 at 13:43
Matt, the "whatever" is not helpful ^^ Thanks anyway Sourav Ghosh, I tried without it and the result is still the same — S-K, Jun 05 '16 at 13:48

Dolda2000 · Accepted Answer · 2016-06-05T15:10:56.830

1

Generally, malloc() and friends will not return NULL only because you are out of physical RAM. They normally don't even know how much physical RAM you have, and will just try to get more from the OS, usually using mmap() (or brk(), but that's just a wrapper for mmap()).

mmap(), too, will not return failure just because you're out of physical RAM, but will try to use virtual memory. This is common across UNIX systems, and it's generally not possible to work directly with physical memory instead of virtual memory. The OOM-killer is just Linux' specific implementation of what happens when virtual memory cannot handle the demand for backing store. One method to make the OOM-killer go away is to allocate more swap space (for this and similar reasons, I find that it often is a good idea to keep a lot of swap space around).

The most common case where mmap() and malloc() will return failure is when they cannot handle the allocation for internal reasons, such as being out of virtual address space (which is pretty rare on 64-bit systems).

That being said, there exist mechanisms for dealing more directly with physical RAM if you want to avoid the potential complexities of virtual memory. One POSIX-defined mechanism is mlock(), which will pin a certain amount of allocated RAM to physical pages, which is usually done to avoid the performance and/or security implications of swapping. It is restricted to the superuser and will usually only allow to lock a small amount of memory in total, however. See its manpage for details.

On Linux, you can also tweak the overcommit behavior. I have to admit I've never tried it myself, but behavior #2 (as documented in the link) seems to promise some kind of behavior similar to what you seem to be looking for.

None of these mechanisms are "ordinary", however, so if you're looking for a way to limit your allocations to physical memory in a way that is portable and reproducible on systems that you don't personally manage, you're probably out of luck, quite simply.

edited Jun 05 '16 at 15:10

answered Jun 05 '16 at 14:04

Dolda2000

25,216
4
51
92

`mmap` maps a file to memory. It is certainly not used by `malloc` & co. – too honest for this site Jun 05 '16 at 14:10
@Olaf: It is also used to map anonymous memory. See `MMAP_ANONYMOUS`. – Dolda2000 Jun 05 '16 at 14:13
`malloc` uses `sbrk` and `brk` underneath(See Ch. 8 of K&R) – lost_in_the_source Jun 05 '16 at 14:18
@stackptr: Sure, but on Linux `brk()` is just a wrapper for `mmap()`, so it's more useful to look at the latter for the general behavior. Also, `malloc()` will start using `mmap()` when it's out of data segment, which happens particularly often on 32-bit architectures where mmapped segments abound in the address space. – Dolda2000 Jun 05 '16 at 14:22
@Dolda2000 Doesn't `sbrk` just advance the pointer to the end of the heap? – lost_in_the_source Jun 05 '16 at 14:23
@stackptr: It does manage an internal pointer that it advances, but what it actually does is that it `mmap`s the space between the previous value of the pointer and the new value. – Dolda2000 Jun 05 '16 at 14:24
Sorry, but I doubt that without authoritative references. And apart from mentioning OOM Killer, the question is not specific to Linux. – too honest for this site Jun 05 '16 at 14:57
@Olaf: You can check `mm/mmap.c:do_brk()` in the Linux source code (and perhaps, indeed, begin by noting that `do_brk()` lives in `mmap.c`). My answer also tries to keep somewhat OS-agnostic, but since there is no portable answer, it also tries to give answers relative to OP's actual environment. – Dolda2000 Jun 05 '16 at 15:00
@dolda2000 Thank you for good explanation of this. I will certanly look into it more. I need to initialize as much free memory as I can, can you help me to achieve that? Maybe give me some sort of a pseudo-code or something like that? Thank you again for the explanation, it is very beginner friendly and nicely said. – S-K Jun 05 '16 at 19:43
@S-K: To be honest, that sounds like a mistake to me. Why would you want to allocate "as much memory as possible", instead of managing your data structures depending on the load they need to carry? Either way, the amount of actually "available" memory would depend on what the rest of the system is doing as well, so it is not a constant or consistent quantity. If anything, you can use `/proc/meminfo` on Linux to find out how much physical memory exists in the system, but it seems questionably useful. – Dolda2000 Jun 05 '16 at 19:50
@dolda2000 I need this for a school project --> initialize all the free memory, wait 10 seconds and close the program. It's the only part of the project I can't seem to do, it always kill the process and I just can't seem to make it. I would really appriciate your help, since I'm trying to do this for 2 weeks already and it took me lots of hours of searching, but I still got nowhere... – S-K Jun 05 '16 at 19:54
@dolda2000 I have a project in which I have to solve 27 different problems and this is one of them. I managed to do others well, but this one is giving me headache. (To make clear that I don't want to take the easy way, but I really checked everything before asking here) – S-K Jun 05 '16 at 19:57
@S-K: It's just, given virtual memory, there are several possible metrics of 'free" memory, to begin with, so what is it actually that you're trying to accomplish? What do you mean by "free" memory? That not used for private memory, that not used by private memory or block caches, or all physical memory? Do you want to count physical memory or virtual memory (including swap-space)? Do you want to actually tie the memory up, or allow the OS to swap it? There are a lot of questions like this in a VM system, so the task is not so well-defined as you seem to imagine it to be. – Dolda2000 Jun 05 '16 at 19:58
@dolda2000 I have to sucessfully alocate maximum possible memory (I think that goes for all the memory - including the virtual and all that... Got to make as much GBs as possible ), than I have to initialize it (set it to zero) and wait 10 seconds and close the program. Can you help me please? – S-K Jun 05 '16 at 20:00
@dolda2000 Edit: in the instructions for the project it says I have to successfully allocate "maximum memory on the system" – S-K Jun 05 '16 at 20:06
Let us [continue this discussion in chat](http://chat.stackoverflow.com/rooms/113866/discussion-between-s-k-and-dolda2000). – S-K Jun 05 '16 at 20:12

C When trying to calloc maximum free memory, no NULL return

1 Answers1