Try allocating chunks of memory until you have no free memory:
int alloced = 0;
for(;;)
{
char *alloc = malloc(10*1024*1024); // alloc 10 MB
if(alloc != NULL)
{
alloced += 10;
// edit: you have to memset the memory otherwise the system will give it back to you next time
memset(alloc, 0xab, 10*1024*1024);
printf(" alloced %d MB\n", alloced);
}
}
edit:
I actually tried just right now on a 64 bits linux with 2GB of ram and 3.3GB of swap: the screen has frozen, I could allocate 4950MB of ram, but then the process was killed by the system, and linux fell back on its feet gracefully, so, no, this doesnt work :=)