Understanding memory growth of a Python process (VmRSS vs gc.get_objects)

Question

Without going into algorithmic details, lets just say that my code sequentially processes a list of inputs:

inputs = [2,5,6,7,8,10,12,13,14,15,16,17,18,19,20,21]
for i in inputs:
    process_input(i)

For simplicity, lets consider process_input to be a state-less black-box.

I know that this site is full of questions about finding memory leaks in Python code, but this is not what this question is about. Instead, I'm trying to understand the memory consumption of my code over time and whether it might suffer from leaking memory.

In particular, I'm trying to understand a discrepancy of two distinct indicators of memory usage:

The number of allocated objects (reported by gc.get_objects) and
the actually used amount of physical memory (read from VmRSS on a Linux system).

To study these two indicators, I expanded the original code from above as follows:

import time, gc

def get_current_memory_usage():
    with open('/proc/self/status') as f:
        memusage = f.read().split('VmRSS:')[1].split('\n')[0][:-3]
    return int(memusage.strip()) / (1024 ** 2)

inputs = [2,5,6,7,8,10,12,13,14,15,16,17,18,19,20,21]
gc.collect()

last_object_count = len(gc.get_objects())

for i in inputs:

    print(f'\nProcessing input {i}...')
    process_input(i)

    gc.collect()
    time.sleep(1)
    memory_usage = get_current_memory_usage()
    object_count = len(gc.get_objects())
    print(f'Memory usage: {memory_usage:.2f} GiB')
    print(f'Object count: {object_count - last_object_count:+}')
    last_object_count = object_count

Note that process_input is state-less, i.e. the order of the inputs does not matter. Thus, we would expect both indicators to be about the same before running process_input and afterwards, right? Indeed, this is what I observe for the number of allocated objects. However, the consumption of memory grows steadily:

Now my core question: Do these observations indicate a memory leak? To my understanding, memory leaking in Python would be indicated by a growth of allocated objects, which we do not observe here. On the other hand, why does the memory consumption grow steadily?

For further investigation, I also ran a second test. For this test, I repeatedly invoked process_input(i) using a fixed input i (five times each) and recorded the memory consumption in between of the iterations:

For i=12, the memory consumption remained constant at 10.91 GiB.
For i=14, the memory consumption remained constant at 7.00 GiB.

I think, these observations make the presence of a memory leak even more unlikely, right? But then, what could be a possible explanation for why the memory consumption is not falling in between of the iterations, given that process_input is state-less?

The system has 32 GiB RAM in total and is running Ubuntu 20.04. Python version is 3.6.10. The process_input function uses several third-party libraries.

score 1 · Answer 1 · answered Jul 14 '20 at 16:38

In general RSS is not a particularly good indicator because it is "resident" set size and even a rather piggish process, in terms of committed memory, can have a modest RSS as memory can be swapped out. You can look at /proc/self/smaps and add up the size of the writable regions to get a much better benchmark.

On the other hand, if there is actually growth, and you want to understand why, you need to look at the actual dynamically allocated memory. What I'd suggest for this is using https://github.com/vmware/chap

To do this, just make that 1 second sleep a bit longer, put a print just before the call to sleep, and use gcore from another session to gather a live core during a few of those sleeps.

So lets say you have cores gathered from when the input was 14 and when it was 21. Look at each of the cores using chap, for example, with the following commands:

count used

That will give you a good view of allocations that have been requested but not released. If the numbers are much larger for the later core, you probably have some kind of growth issue. If those numbers do differ by quite a lot, use

summarize used

If you have growth, it is possible that there is a leak (as opposed to some container simply expanding). To check this, you can try commands like

count leaked
show leaked

From there you should probably look at the documentation, depending on what you find.

OTOH if used allocations are not the issue, maybe try the following, to see memory for allocations that have been released but are part of larger regions of memory that cannot be given back to the operating system because parts of those regions are still in use:

count free
summarize free

If neither "used" allocations or "free" allocations are the issue, you might try:

summarize writable

That is a very high level view of all writable memory. For example, you can see things like stack usage...

Understanding memory growth of a Python process (VmRSS vs gc.get_objects)

1 Answers1