Fail to malloc big block memory after many malloc/free small blocks memory

Question

Here is the code.

First I try to malloc and free a big block memory, then I malloc many small blocks memory till it run out of memory, and I free ALL those small blocks.

After that, I try to malloc a big block memory.

#include <stdio.h>
#include <stdlib.h>
int main (int argc, char **argv)
{
    static const int K = 1024;
    static const int M = 1024 * K;
    static const int G = 1024 * M;

    static const int BIG_MALLOC_SIZE = 1 * G;
    static const int SMALL_MALLOC_SIZE = 3 * K;
    static const int SMALL_MALLOC_TIMES = 1 * M;

    void **small_malloc = (void **)malloc(SMALL_MALLOC_TIMES * sizeof(void *));

    void *big_malloc = malloc(BIG_MALLOC_SIZE);
    printf("big malloc first time %s\n", (big_malloc == NULL)? "failed" : "succeeded");
    free(big_malloc);

    for (int i = 0; i != SMALL_MALLOC_TIMES; ++i)
    {
        small_malloc[i] = malloc(SMALL_MALLOC_SIZE);
        if (small_malloc[i] == NULL)
        {
            printf("small malloc failed at %d\n", i);
            break;
        }
    }
    for (int i = 0; i != SMALL_MALLOC_TIMES && small_malloc[i] != NULL; ++i)
    {
        free(small_malloc[i]);
    }

    big_malloc = malloc(BIG_MALLOC_SIZE);
    printf("big malloc second time %s\n", (big_malloc == NULL)? "failed" : "succeeded");
    free(big_malloc);

    return 0;
}

Here is the result:

big malloc first time succeeded
small malloc failed at 684912
big malloc second time failed

It looks like there are memory fragments.

I know memory fragmentation happens when there are many small empty space in memory but there is no big enough empty space for big size malloc.

But I've already free EVERYTHING I malloc, the memory should be empty.

Why I can't malloc big block at the second time?

I use Visual Studio 2010 on Windows 7, I build 32-bits program.

Perhaps the memory is still fragmented. Have you tried to monitor what happens to memory while your program is running? — Ashalynd, May 28 '14 at 09:30
Note: You are freeing uninitialized pointers (after a small malloc fail) — , May 28 '14 at 09:30
@DieterLücking I thought so too, but the `free()`-loop ends when the first `NULL` is encountered. — unwind, May 28 '14 at 09:32
Although I'm not sure it's enough to fix this code, when VC++ has problems due to heap fragmentation, you can *sometimes* help by calling `_heapmin()`. — Jerry Coffin, May 28 '14 at 09:32
@DieterLücking When smalloc malloc fail, malloc return NULL. In the second for loop I check small_malloc[i] != NULL to guard the uninitialized pointers. — Celebi, May 28 '14 at 09:34
Obviously, VC++ fails to re-combine adjacent free memory blocks. With GCC on Linux 32 bit, it works. — glglgl, May 28 '14 at 09:37
The problem is not VC++ but libc's heap manager. By default, windows' heap manager is really conservative, but there's an API to request the use of the new heap manager — toasted_flakes, May 28 '14 at 09:39
Any chance it is due to using of the debug version of C runtime library? — 9dan, May 28 '14 at 09:45
@Celebi use `size_t` not `int` for storing your malloc values, `int` is not capable of holding the size of allocable memory on the system, and its also signed so you risk overflow. — Mgetz, May 28 '14 at 18:34

score 4 · Accepted Answer · answered May 28 '14 at 17:58

The answer, sadly, is still fragmentation.

Your initial large allocation ends up tracked by one allocation block; however when you start allocating large numbers of 3k blocks of memory your heap gets sliced into chunks.

Even when you free the memory, small pieces of the block remain allocated within the process's address space. You can use a tool like Sysinternals VMMap to see these allocations visually.

It looks like 16M blocks are used by the allocator, and once these blocks are freed up they never get returned to the free pool (i.e. the blocks remain allocated).

As a result you don't have enough contiguous memory to allocate the 1GB block the second time.

score 0 · Answer 2 · edited May 23 '17 at 10:32

0

Even I know just a little about this, I found the following thread Why does malloc not work sometimes? which covers the similar topic as yours. It contains the following links: http://www.eskimo.com/~scs/cclass/int/sx7.html (Pointer Allocation Strategies) http://www.gidforums.com/t-9340.html (reasons why malloc fails? )

edited May 23 '17 at 10:32

Community

1
1

answered May 28 '14 at 10:06

Jaroslav

1

1

just a comment: I tried your code under Win 7 32 bit/4GB built in CVI 2013 and there is no problem as you described, so it is probably VC2010 (or itssub-part) specific. – Jaroslav May 28 '14 at 10:30

score 0 · Answer 3 · answered May 28 '14 at 18:46

The issue is likely that even if you free every allocation, malloc does not return all the memory to the operating system.

When your program requested the numerous smaller allocations, malloc had to increase the size of the "arena" from which it allocates memory.

There is no guarantee that if you free all the memory, the arena will shrink to the original size. It's possible that the arena is still there, and all the blocks have been put into a free list (perhaps coalesced into larger blocks).

The presence of this lingering arena in your address space may be making it impossible to satisfy the large allocation request.

Fail to malloc big block memory after many malloc/free small blocks memory

3 Answers3