2

I'm using CentOS 7 and I'm running a C++ Application. Recently I switched to a newer version of a library which the application was using for various MySQL C API functions. But after integrating the new library, I saw a tremendous increase in memory usage of the program i.e. the application crashes if left running for more than a day or two. Precisely, what happens is the memory usage for the application starts increasing upto a point where the application alone is using 74.9% of total memory of the system and then it is forcefully shut down by the system.

Is there any way of how to track memory usage of the whole application including the static variables as well. I've already tried valgrind's tool Massif.

Can anyone tell me what could be the possible reasons for the increased memory usage or any tools that can give me a deep insight of how the memory is being allocated (both static and dynamic). Is there any tool which can tell us about Memory Allocation for a C++ Application running in a linux environment?

Thanks in advance!

Abhinav
  • 514
  • 1
  • 7
  • 17
  • 2
    Valgrind should be the right tool for analysis of this kind of problem, which seems to be a classic memory leak. – Torbjörn Aug 19 '16 at 09:10
  • @Torbjörn, I've checked with valgrind as well and the memory allocation difference in the HEAP SUMMARY section that comes up for the execution of the application is around 1 GB for runtime durations of around 10 mins. – Abhinav Aug 19 '16 at 09:15
  • You could also override the global new operator and delete operators. I have done this before to keep a cache of heap allocated memory pointers that are not yet deleted. Then a separate thread that prints that information out every few minutes. – Dennis Aug 19 '16 at 09:15

1 Answers1

1

Static memory is allocate when the program starts. Are you seeing memory growth or a startup increase?

Since it takes 'a day or two to crash', the trouble is likely a memory leak or unbounded growth of a data structure. Valgrind should be able to help with both. If valgrind shows a big leak with the --leak-check-full option then you will likely have found the issue.

To check for unbounded growth, put a preemptive _exit() in the program at a point where you suspect the heap has grown. For example, put a timer on the main loop and have the program _exit after 10 minutes. If the valgrind shows a big 'in use at exit' then you likely have unbounded growth of a data structure but not a leak. Massif can help track this down. The ms_print gives details of allocations with function stack.

If you find an issue, try switching back to the older version of your library. If the problem goes away, check and make sure you are using the API properly in the new version. If you don't have the source code then you are a bit stuck in terms of a fix.

If you want to go the extra mile, you can write a shared library interposer for malloc/free to see what is happening. Here is a good start. Linux has the backtrace functionality that can help with determining the exact stack.

Finally, if you must use the 3rd party library and find the heap growing without bound or leaking then you can use the shared library interposer to directly call free/delete. This is a risky last-ditch unrecommended strategy but I've used in production to limp a process along.

Community
  • 1
  • 1
Matthew Fisher
  • 2,258
  • 2
  • 14
  • 23