1

We have a program that crashes at 4Gb due to insufficient memory after running for a long period of time. The project is in C++, openSuse11,12, including linking to various shared memories and other libraries.

Monitoring the heap increase and total memory increase from the pmap of the PID, they are totally analogous. Meaning the process starts with 2.5Gb and heap ~0 , and before crashing Total = 4 Gb and heap = 1.5 Gb.

We have made some runs with valgrind and the result is quite puzzling. Below you can see the summaries for running 20 minutes. 

==30042== HEAP SUMMARY:
==30042==     in use at exit: 706,413 bytes in 3,345 blocks
==30042==   total heap usage: 1,770,240 allocs, 1,766,895 frees, 173,813,303 bytes allocated

==30042== LEAK SUMMARY:
==30042==    definitely lost: 96 bytes in 3 blocks
==30042==    indirectly lost: 27 bytes in 3 blocks
==30042==      possibly lost: 344 bytes in 2 blocks
==30042==    still reachable: 705,946 bytes in 3,337 blocks
==30042==                       of which reachable via heuristic:
==30042==                         stdstring          : 62 bytes in 2 blocks
==30042==         suppressed: 0 bytes in 0 blocks

Monitoring the pmap, we know that the total memory increased by 130 Mb but comparing this to the valgrind report, we can see there are leaks but nothing really close to the 130 Mb.

Now to the questions:

  • Can someone help me understand the valgrind summary and how it can relate to the 130 Mb lost ?

  • Is this a leak or not ?

  • Does valgrind report what is going on inside the shared memories ?
  • Even if there is a leak in the SM, wouldn't it still be in heap in them? Their sizes are in reasonable levels.
  • I have read in various sources, that even when freeing the memory with delete[] or free() the system still may keep the space reserved for future use. If that is the case, what can be done ?

*EDIT The place this heap usage seems to be unrestricted is when the program which is a server, makes calls to a DB (solid SQL) for large data continuously, which allocates a lot of space. It is not in a shared memory. And we see that frees are correctly managed, what can be done ?

thahgr
  • 718
  • 1
  • 10
  • 27
  • 1
    Concerning the use of `free`: When you malloc an object, your program makes a request to the OS for more memory pages if need be. What you have been reading is that `free` doesn't release those pages back to the OS, it keeps them around. This means that when you request more memory again, the program doesn't need to ask the OS for more pages: it already has those pages. Your program will overwrite the old data on free'd memory, so this isn't an actual leak. – CoconutBandit Sep 22 '16 at 14:54
  • Most of it is reachable, meaning that the variables pointing to it are not out of scope or not destroyed. Which means that it could be legitimately used memory. – imreal Sep 22 '16 at 14:56
  • Are you on a 32 bit environment? – imreal Sep 22 '16 at 14:58
  • The *still reachable: 705,946 bytes in 3,337 blocks* is stuff that should get cleaned up at the end. see: http://stackoverflow.com/questions/30376601/valgrind-memory-still-reachable-with-trivial-program-using-iostream – NathanOliver Sep 22 '16 at 15:02
  • @imreal its a 64bit environment – thahgr Sep 23 '16 at 08:30

1 Answers1

1

Take a look at the Massif tool in the Valgrind suite. The heap profiler will help you understand how the program continues to grow without bound. Unlimited growth appears to be the problem rather than leaking.

Can someone help me understand the valgrind summary and how it can relate to the 130 Mb lost ?

Valgrind indicates the program used 170mb of heap which is not too far from your number. Longer runs are likely to show an ever increasing maximum heap size.

Is this a leak or not ?

The report does not indicate much in the way of leaks. It appears that the program cleans up pretty well. The peak heap usage seems to be issue.

Does valgrind report what is going on inside the shared memories ?

No, Valgrind looks at heap memory which is process local.

Even if there is a leak in the SM, wouldn't it still be in heap in them? Their sizes are in reasonable levels.

No, shared memory is not heap memory. ipcs and other tools can show you the amount of shared memory being used. The total addressable space of a 32-bit process is 4gb so the amount of shared memory mapped into the process can reduce the amount of heap space available.

I have read in various sources, that even when freeing the memory with delete[] or free() the system still may keep the space reserved for future use. If that is the case, what can be done ?

Yes this is accurate. Once memory is allocated to a process it will not be returned to the OS until the process exits. The heap will be reused repeatedly in a process so the maximum size of the heap will not increase unless actually in use. Nothing can done at the user level and generally this is not a problem.

To reiterate, likely a data structure in the program is growing without bound. For example, a linked list that is being endlessly added to. Since the data is still reachable, this is not technically a leak and will not show in the valgrind report. A heap profiler, like Massif, can point you to areas in the source code that allocate large amounts of heap memory.

Matthew Fisher
  • 2,258
  • 2
  • 14
  • 23
  • Thanks for your answer! The program is a server app which must be always on, how can someone cope with the unlimited growth you mention ? Since it seems to clean up pretty well, what can someone do besides restarting the app ? – thahgr Sep 23 '16 at 08:35
  • 1
    There only 3 options. 1) Restart the process before it runs out of memory. 2) Switch to a 64-bit model so the process can continue to grow. 3) Understand the memory usage better and release unused memory back to the heap. 3) is the best but may also be the hardest. 1) is simple and may be the most immediately useful in a production environment. – Matthew Fisher Sep 23 '16 at 14:15