1

I have a process which continuously allocates memory and will free it after another thread have processed related data. When the data processing rate is slow, I see RES memory grows up; but after all the data have been processed, RES goes down but doesn't go back to original RES value (even after waiting for over 10 minutes).

e.g. 10 MB (original) => 50 MB (peak) => 30MB (after all data are freed)

I have used valgrind with massif to profile memory, it looks like all data are freed. My question is why RES doesn't go back to original 10 MB?

gsamaras
  • 71,951
  • 46
  • 188
  • 305
kai
  • 1,141
  • 3
  • 15
  • 25
  • This is because it is on the disposition of the memory allocator when exactly memory on the heap is returned to the OS. But you have the option to manage the memory allocations yourself with `mmap()/munmap()` to be in full control over this. – Ctx Nov 27 '17 at 10:08
  • C or c++? I.e. `new` or `malloc`? – MSalters Nov 27 '17 at 12:50

2 Answers2

1

There are so many possible reasons, I'll just list you the two most common ones:

  • page-wise allocation and fragmentation: Operating systems allocate memory in terms of pages, at least on modern systems using MMUs. Your standard library's malloc() (or the allocator for new in C++) serves arbitrary sizes and manages them internally. A page can only be returned to the operating system when the last allocation occupying it is returned. If only a single allocation in the page is still active, your process must keep this page.

  • libraries: A lot of libraries do their own dynamic memory allocations, even the C standard library does.

That said, as Ctx commented, you can do direct page allocations yourself on Linux using mmap() and maybe avoid the fragmentation issue by using these pages efficiently. This would of course mean to leave the path of standard C, but at least, mmap() is specified in POSIX, so it would work on many systems.

1

There are many examples of OSes that do not give the memory back to the OS when free() is called.

However, when malloc() or realloc() is called, it will see that free'd memory as available.

Perfomance and paging is the reasoning behind this. Read more in Memory usage doesn't decrease when free() used.

That's explains what you witness (the memory usage outside of your program not to be decreased, even though you are free-ing your memory; because it might be that it will be marked free inside your program's scope only).

gsamaras
  • 71,951
  • 46
  • 188
  • 305