2

I have a Linux/Mac C++ library that performs a series of steps, in the form of function calls. I would like to measure the maximum memory usage during each step.

I am not interested in ad-hoc solutions like starting another thread that polls memory usage, running a profiler, etc.

So far I have found getrusage() is present on Mac and Linux and does return the maximum memory usage, however there appears to be no way to reset this maximum after each function call.

Is there any way around this limitation?

Edit: To be clear, I do not want to commandeer malloc()/free() and log everything. I want a solution that is suitable to keep in running production code.

Timmmm
  • 88,195
  • 71
  • 364
  • 509
  • https://stackoverflow.com/questions/4690800/how-to-profile-memory-usage – Mat Jan 21 '19 at 10:12
  • That is about adding profilers to my program which I stated I do not want to do. – Timmmm Jan 21 '19 at 10:16
  • Why adding? The second one is a precompiled package that is preloaded. – Matthieu Brucher Jan 21 '19 at 10:16
  • Added heaptrack in the linked question. – Matthieu Brucher Jan 21 '19 at 10:20
  • This is a library that will be distributed to other people. I don't want to force a profiler like `gperftools` on them. It isn't designed to be run in production use - only for debugging. – Timmmm Jan 21 '19 at 10:22
  • Linux and Mac OS are very different. Writing "Linux/Mac" doesn't magically make them similar. If against all odds there is a solution applicable to both platforms, congrats, you have won the lottery. – n. m. could be an AI Jan 21 '19 at 10:39
  • They *are* similar (e.g. they both have `getrusage()`). I never said they were identical or that any solution would work the same on both OSes. I am ok with `#ifdef __linux__`. – Timmmm Jan 21 '19 at 11:18
  • "they both have getrusage()". getrusage is POSIX standard, ru_maxrss is not. Both OSes happen to have ru_maxrss but the meaning is not the same (bytes or kilobytes?) and you cannot expect *anything* related to ru_maxrss to be the same. – n. m. could be an AI Jan 23 '19 at 11:20
  • I said they were ***similar*** not identical. – Timmmm Jan 23 '19 at 11:47

2 Answers2

1

I had a browse through the Linux source code, and found this:

        /*
         * Writing 5 to /proc/pid/clear_refs resets the peak
         * resident set size to this mm's current rss value.
         */

I haven't tried it yet but it looks promising.

Edit: It was added in this commit

Edit 2: I have looked through the MacOS kernel source - the corresponding value is stored in resident_max. Unfortunately there doesn't seem to be a feature to reset it.

Edit 3: On Linux you can obtain the maximum allocated memory using malloc_info() however there does not appear to be a way to reset it. It also relies on you using glibc.

Timmmm
  • 88,195
  • 71
  • 364
  • 509
0

The malloc and free calls are more than trivial wrappers around sbrk and mmap system calls. This makes getrusage return something that is not in line with calls to malloc and free. Any non-trivial implementation of these functions will manage a free-list inside the process itself, before thinking of returning anything to the system.

The program calls free (or delete for that matter), and the memory is not immediately returned to the operating system (maybe never). The free-ed memory can be reused by the task, if it calls malloc, but not by other processes. This makes getrusage correct from the OS perspective, but not correct from the program perspective.

On Linux you can use mallinfo()

#include <malloc.h>
#include <cinttypes>
std::size_t memsize()
{
    auto data = mallinfo();
    return data.arena - data.fordblks + data.hblkhd;
}

Here, memsize() will return the number of bytes allocated from program's perspective. It takes into account the various allocation techniques, such as sbrk and mmap, and considers rounding up and overhead as part of the allocated memory of malloc() (and new).

With OSX things are not so bright. You can have a look into the source code of apple's malloc(), and specifically at mstats, which states in the comment:

/*
 * A Glibc-like mstats() interface.
 *
 * Note that this interface really isn't very good, as it doesn't understand
 * that we may have multiple allocators running at once.  We just massage
 * the result from malloc_zone_statistics in any case.
 */

This does not look very promising and:

#include <malloc/malloc.h>
#include <cinttypes>
std::size_t memsize()
{
    auto data = mstats();
    return data.bytes_used;
}

Does not look very good, according to my experiments. But it may be better than nothing, or just relying on getrusage. I think you are out of luck on OSX, unless someone can correct me.

Michael Veksler
  • 8,217
  • 1
  • 20
  • 33
  • This is all about getting the *current* memory usage right? I want the *peak* usage. Useful information nonetheless. – Timmmm Jan 21 '19 at 13:12
  • You can know it! Both Linux and Mac record this information (in terms of RSS which would be fine for my purposes). Additionally `malloc_info()` returns a `maximum` field which may be what I want but it is completely undocumented so I'm not sure yet. – Timmmm Jan 21 '19 at 13:33
  • I don't know what you mean by "`getrusage` seems to be an approximation". I'm pretty sure it isn't. – Timmmm Jan 21 '19 at 13:35
  • @Timmmm forget about what I wrote, a mistake – Michael Veksler Jan 21 '19 at 13:40