memory leak analyse with top

Question

I would like to use the tool "top" to analyze the memory consumption and possible memory leaks of a process. For this I have written this program (program-name: memoryTest):

int main(){
char* q;
for(int i=0; i<100; i++){
    q = (char*) malloc(1024); 
    sleep(1);
}
return 0;

}

With top I can now watch this program, by filtering with the option "o" and the filter specification "COMMAND = memoryTest" after the said process, however I see no change in the memory consumption of the process. Do I have a stupid mistake here?

Your runtime/OS is probably being clever and not actually allocating memory you requested but don't use. Try writing stuff into the memory. — nwp, Dec 05 '18 at 13:59
Your compiler does the optimization and malloc calls were stripped out from the final code. — tunglt, Dec 05 '18 at 14:18
@TungLeThanh I am not entirely sure whether this is a valid optimization. You can, even at runtime, "redirect" `malloc` calls to your own function with side effects (e.g., by `LD_PRELOAD`), which compiler can't know at compile time. Related discussion: https://stackoverflow.com/q/53373421/580083 — Daniel Langr, Dec 05 '18 at 14:43
@DanielLangr: I observed the behavior when compiling the given application with -O1 activated (by gcc). — tunglt, Dec 05 '18 at 14:50
@TungLeThanh You are right, the rationale is that `malloc` is not an external function, but a library function with no side effects. Thus the compiler is allowed to optimize out its calls. — Daniel Langr, Dec 05 '18 at 14:53
You can see the change in the memory comsumption if you change the memory request size to 1024*1024. In fact, glibc use about 128K (or 1M, depended on model) blocks for memory allocator. If your request size is too small, your application will use the avails memory until the 128K block is full. — tunglt, Dec 05 '18 at 14:54

tunglt · Answer 1 · 2018-12-06T12:17:28.180

From malloc man page :

Normally, malloc() allocates memory from the heap, and adjusts the size of the heap as required, using sbrk(2). When allocating blocks of memory larger than MMAP_THRESHOLD bytes, the glibc malloc() implementation allocates the memory as a private anonymous mapping using mmap(2). MMAP_THRESHOLD is 128 kB by default, but is adjustable using mallopt(3). Prior to Linux 4.7 allocations performed using mmap(2) were unaffected by the RLIMIT_DATA resource limit; since Linux 4.7, this limit is also enforced for allocations performed using mmap(2).

The memory pools are called arenas and the implementation is in arena.c. The macro HEAP_MAX_SIZE define the maximum size of an arena and it is basically 1MB on 32-bit and 64MB on 64-bit:

HEAP_MAX_SIZE = (2 * DEFAULT_MMAP_THRESHOLD_MAX)
32-bit [DEFAULT_MMAP_THRESHOLD_MAX = (512 * 1024)] = 1,048,576 (1MB)
64-bit [DEFAULT_MMAP_THRESHOLD_MAX = (4 * 1024 * 1024 * sizeof(long))] = 67,108,864 (64MB)

Information from heap implementation (arena.c):

/* A heap is a single contiguous memory region holding (coalesceable) malloc_chunks. It is allocated with mmap() and always starts at an address aligned to HEAP_MAX_SIZE. */

EDIT:

Heap allocation can be observed by using strace. In the first call to brk(), the main arena is allocated with 200K bytes (72K from libstdc++ with 128K top_pad ).

brk(NULL)                               = 0x556ecb423000 -> current program break
brk(0x556ecb455000)                     = 0x556ecb455000 -> resize the heap by moving brk 0x32000 bytes upward (main arena initialization with 200K). 
write(1, "i = 0\n", 8)                = 8
...
write(1, "i = 123\n", 8)                = 8     
brk(0x556ecb476000)                     = 0x556ecb476000 -> resize the heap by moving brk 0x21000 bytes upward (growing heap 128K). 
...
write(1, "i = 252\n", 8)                = 8
brk(0x556ecb497000)                     = 0x556ecb497000 -> resize the heap by moving brk 0x21000 bytes upward (growing heap 128K).

Your application used only 100K bytes of 128K available heap, so the memory consumption would not be observed by top or htop program.

You can see the change in memory consumption easily if you force the glibc uses mmap() by requesting blocks larger than 128K or by increasing the number of blocks ( > 128 ).

Owl · Answer 2 · 2018-12-05T15:40:37.207

Try the following:

#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <unistd.h>

int main(){
  char* q;
  for(int i=0; i<10000; i++){
    q = (char*)malloc(1024); 
    memset(q,0,1024);
  }
  getchar();
  return 0;
}

for different values in the for loop.

The issue here is Linux doesn't necessary use memory even if it's allocated until it's actually populated. Therefore you need to write data to what you're allocating else it might not even register that that memory is in use. This is why some applications can allocate memory and it's fine, then even when they have allocated, when they come to use the memory, they can find that the memory isn't available.

The memset will force writes of zeros into the allocated buffer, thereby causing the memory to be used and register as being used in top. Note that htop may be easier for you to use here.

If you want to follow up further look into "optimistic malloc" which is a Linux characteristic, note that some other operating systems do not behave this way.

Also it's worth pointing out that internally memory is allocated in contiguous chunks of a certain size. So allocating, say an additional 1KB may not register a size increase if the memory is allocated in minimal block size of say 4k.

score 0 · Answer 3 · answered Dec 06 '18 at 00:43

If you want to understand the memory usage, don't use top. Use the free open source program https://github.com/vmware/chap (disclaimer: I am the original developer).

Just gather a live core before the program finishes, for example by running gcore, then type:

chap your-core-file-name

Then you can do things like:

count leaked list leaked count used count free ....

Here is an example using your program, gathering the core after 30 seconds:

-bash-4.1$ ./q53633998 &
[1] 18014
-bash-4.1$ sleep 30
gcore 18014
-bash-4.1$ gcore 18014
0x00000030ed6aca20 in __nanosleep_nocancel () at ../sysdeps/unix/syscall-         template.S:82
82      T_PSEUDO (SYSCALL_SYMBOL, SYSCALL_NAME, SYSCALL_NARGS)
Saved corefile core.18014
-bash-4.1$ ~/public_html/chap core.18014
chap> count used
36 allocations use 0x9120 (37,152) bytes.
chap> count free
1 allocations use 0x17db0 (97,712) bytes.
chap> count leaked
35 allocations use 0x8d18 (36,120) bytes.
chap> count anchored
1 allocations use 0x408 (1,032) bytes.
chap> list anchored
Used allocation at 19d4e40 of size 408

1 allocations use 0x408 (1,032) bytes.
chap> explain 19d4e40
Address 19d4e40 is at offset 0 of
an anchored allocation at 19d4e40 of size 408
Allocation at 19d4e40 appears to be directly anchored from at least one stack.
Address 0x7ffc88570270 is on the live part of the stack for thread 1.
Stack address 7ffc88570270 references 19d4e40

chap> list leaked
Used allocation at 19cc010 of size 408

Used allocation at 19cc420 of size 408

Used allocation at 19cc830 of size 408

Used allocation at 19ccc40 of size 408

Used allocation at 19cd050 of size 408

Used allocation at 19cd460 of size 408

Used allocation at 19cd870 of size 408

Used allocation at 19cdc80 of size 408

Used allocation at 19ce090 of size 408

Used allocation at 19ce4a0 of size 408

Used allocation at 19ce8b0 of size 408

Used allocation at 19cecc0 of size 408

Used allocation at 19cf0d0 of size 408

Used allocation at 19cf4e0 of size 408

Used allocation at 19cf8f0 of size 408

Used allocation at 19cfd00 of size 408

Used allocation at 19d0110 of size 408

Used allocation at 19d0520 of size 408

Used allocation at 19d0930 of size 408

Used allocation at 19d0d40 of size 408

Used allocation at 19d1150 of size 408

Used allocation at 19d1560 of size 408

Used allocation at 19d1970 of size 408

Used allocation at 19d1d80 of size 408

Used allocation at 19d2190 of size 408

Used allocation at 19d25a0 of size 408

Used allocation at 19d29b0 of size 408

Used allocation at 19d2dc0 of size 408

Used allocation at 19d31d0 of size 408

Used allocation at 19d35e0 of size 408

Used allocation at 19d39f0 of size 408

Used allocation at 19d3e00 of size 408

Used allocation at 19d4210 of size 408

Used allocation at 19d4620 of size 408

Used allocation at 19d4a30 of size 408

35 allocations use 0x8d18 (36,120) bytes.
chap> list free
Free allocation at 19d5250 of size 17db0

1 allocations use 0x17db0 (97,712) bytes.
chap>

That last "free" allocation is the tail of a block of memory that was allocated during the first malloc call and gradually carved up as subsequent malloc calls were made.

Of course there are other tools out there (e.g. valgrind) that work differently by instrumenting the process but if you want a tool that can analyze memory usage of a process without changing the way the process is run, chap is a good choice.

memory leak analyse with top

3 Answers3