Why is previously allocated memory inaccessable after a failed realloc?

Question

I have a need to allocate all memory available to a process, in order to implement a test of a system service. The test (among others) requires exhausting all available resources, attempting the call, and checking for a specific outcome.

In order to do this, I wrote a loop that reallocates a block of memory until, realloc returns null, then using the last good allocation, then cutting the difference between the last successful quantity and the last unsuccessful quantity until the unsuccessful quantity is 1 byte larger than the last successful quantity, guaranteeing that all available memory is consumed.

The code I wrote is as follows (debug prints also included)

#include <stdio.h>
#include <malloc.h>
int main(void)
{
    char*       X;
    char*       lastgood = NULL;
    char*       toalloc = NULL;
    unsigned int    top = 1;    
    unsigned int    bottom = 1; 
    unsigned int    middle;     
    do              
    {               
        bottom = top;
        lastgood = toalloc;     
        top = bottom*2;                 
        printf("lastgood = %p\ntoalloc = %p\n", lastgood, toalloc); 
        if (lastgood != NULL)           
            printf("*lastgood = %i\n", *lastgood);      
        toalloc = realloc(toalloc, top);    
        printf("lastgood = %p\ntoalloc = %p\n", lastgood, toalloc); 
        if (toalloc == NULL && lastgood != NULL)        
            printf("*lastgood = %i\n", *lastgood);  //segfault happens here 
    }while(toalloc != NULL);                    
    do                          
    {                           
        if (toalloc != NULL) lastgood = toalloc;        
        else toalloc = lastgood;                
        middle = bottom+(top - bottom)/2;           
        toalloc = realloc(toalloc, middle);         
        if (toalloc == NULL) top = middle;          
        else bottom = middle;               
    }while(top - bottom > 1);               
    if (toalloc != NULL) lastgood = toalloc;                
        X = lastgood;
//make a call that attempts to get more memory
    free(X);
}

According to realloc's manpage, realloc does not destroy the previous address if it returns null. Even so, this code results in a segfault when it tries to print lastgood after toalloc receives NULL from realloc. Why is this happening, and is there a better way to just grab the exact quantity of unallocated memory?

I am running it on glibc, on ubuntu with kernel 3.11.x

Answer 1

You are not checking the value of top for overflow. This is what happens with its value:

2
4
8
16
32
64
128
256
512
1024
2048
4096
8192
16384
32768
65536
131072
262144
524288
1048576
2097152
4194304
8388608
16777216
33554432
67108864
134217728
268435456
536870912
1073741824
2147483648
0

Just before the last realloc(), the new value of top is 0 again (actually 2^32 but that doesn't fit in 32 bits), which seems to cause the memory block to actually deallocate.

---

Trying to allocate the maximum contiguous block is not a good idea. The memory map seen by the user process has some block already allocated for shared libraries, and the actual code and data of the current process. Unless you want to know the maximum contiguous memory block you can allocate, the way to go is to allocate as much as you can in a single block. When you have reached that, do the same with a different pointer, and keep doing that until you really run out of memory. Take into account that in 64-bit systems, you don't get all available memory in just one malloc()/realloc(). As I've just seen, malloc() in 64-bit systems allocate up to 4GB of memory in one call, even when you can issue several mallocs() and still succeed on each of them.

A visual of a user process memory map as seen in a 32-bit Linux system is described in an answer I gave a few days ago: Is kernel space mapped into user space on Linux x86?

I've come up with this program, to "eat" all the memory it can:

#include <stdio.h>
#include <malloc.h>

typedef struct slist
{
  char *p;
  struct slist *next;
} TList;

int main(void)
{
  size_t nbytes;
  size_t totalbytes = 0;
  int i = 0;
  TList *list = NULL, *node;

  node = malloc (sizeof *node);
  while (node)
  {
    node->next = list;
    list = node;
    nbytes = -1; /* can I actually do this? */ 
    node->p  = malloc(nbytes);  
    while (nbytes && !node->p)
    {
      nbytes/=2;
      node->p = malloc(nbytes);
    }
    totalbytes += nbytes + sizeof *node;
    if (nbytes==0)
      break;
    i++;
    printf ("%8d", i);
  }
  printf ("\nBlocks allocated: %d. Memory used: %f GB\n", 
                 i, totalbytes/(1024*1048576.0));
  return 0;
}

The execution yields these values in a 32-bit Linux system:

 1       2       3       4       5       6       7       8       9      10
11      12      13      14      15      16      17      18      19      20
21      22      23      24      25      26      27      28      29      30
31      32      33      34      35      36      37      38      39      40
41      42      43      44      45      46      47      48      49      50
51      52      53
Blocks allocated: 53. Memory used: 2.998220 GB

Very close to the 3GB limit on 32-bit Linux systems. On a 64-bit Linux system, I've reached 30000 blocks of 4GB each, and still counting. I don't really know if Linux can allocate that much memory or it's my mistake. According to this, the maximum virtual address space is 128TB (that would be 32768 4GB blocks)

---

UPDATE: in fact, so it is. I've left running this program on a 64-bit box, and after 110074 succesuflly allocated blocks, the grand total of memory allocated has been 131071.578884 GB . Each malloc() has been able to allocate as much as 4 GB per operation, but when reached 115256 GB, it has begun to allocate up to 2 GB, then when it reached 123164 GB allocated, it begun to allocate up to 1GB per malloc(). The progression asyntotically tends to 131072 GB, but it actually stops a little earlier, 131071.578884 GB because the very process, its data and shared libraries, uses a few KB of memory.

Answer 2

I think getrlimit() is the answer to your new question. Here is the man page http://linux.die.net/man/2/getrlimit

You are probably interested in RLIMIT_STACK. Please check the man page.

Answer 3

first part is replaced with the following.

char *p;
size_t size = SIZE_MAX;//SIZE_MAX : #include <stdint.h>
do{
    p = malloc(size);
    size >>= 1;
}while(p==NULL);