1

I have written an application with flask and uses celery for a long running task. While load testing I noticed that the celery tasks are not releasing memory even after completing the task. So I googled and found this group discussion..

https://groups.google.com/forum/#!topic/celery-users/jVc3I3kPtlw

In that discussion it says, thats how python works.

Also the article at https://hbfs.wordpress.com/2013/01/08/python-memory-management-part-ii/ says

"But from the OS’s perspective, your program’s size is the total (maximum) memory allocated to Python. Since Python returns memory to the OS on the heap (that allocates other objects than small objects) only on Windows, if you run on Linux, you can only see the total memory used by your program increase."

And I use Linux. So I wrote the below script to verify it.

import gc
def memory_usage_psutil():
    # return the memory usage in MB
    import resource
    print 'Memory usage: %s (MB)' % (resource.getrusage(resource.RUSAGE_SELF).ru_maxrss / 1000.0)

def fileopen(fname):
    memory_usage_psutil()# 10 MB
    f = open(fname)
    memory_usage_psutil()# 10 MB
    content = f.read()
    memory_usage_psutil()# 14 MB

def fun(fname):
    memory_usage_psutil() # 10 MB
    fileopen(fname)
    gc.collect()
    memory_usage_psutil() # 14 MB

import sys
from time import sleep
if __name__ == '__main__':
    fun(sys.argv[1])
    for _ in range(60):
        gc.collect()
        memory_usage_psutil()#14 MB ...
        sleep(1)

The input was a 4MB file. Even after returning from the 'fileopen' function the 4MB memory was not released. I checked htop output while the loop was running, the resident memory stays at 14MB. So unless the process is stopped the memory stays with it.

So if the celery worker is not killed after its task is finished it is going to keep the memory for itself. I know I can use max_tasks_per_child config value to kill the process and spawn a new one. Is there any other way to return the memory to OS from a python process?.

Shinto C V
  • 714
  • 1
  • 9
  • 16

1 Answers1

3

I think your measurement method and interpretation is a bit off. You are using ru_maxrss of resource.getrusage, which is the "high watermark" of the process. See this discussion for details on what that means. In short, it is the peak RAM usage of your process, but not necessarily current. Parts of the process could be swapped out etc.

It also can mean that the process has freed that 4MiB, but the OS has not reclaimed the memory, because it's faster for the process to allocate new 4MiB if it has the memory mapped already. To make it even more complicated programs can and do use "free lists", lists of blocks of memory that are not in active use, but are not freed. This is also a common trick to make future allocations faster.

I wrote a short script to demonstrate the difference between virtual memory usage and max RSS:

import numpy as np
import psutil
import resource


def print_mem():
    print("----------")
    print("ru_maxrss: {:.2f}MiB".format(
            resource.getrusage(resource.RUSAGE_SELF).ru_maxrss / 1024))
    print("virtual_memory.used: {:.2f}MiB".format(
            psutil.virtual_memory().used / 1024 ** 2))


print_mem()
print("allocating large array (80e6,)...")
a = np.random.random(int(80e6))

print_mem()
print("del a")
del a

print_mem()
print("read testdata.bin (~400MiB)")
with open('testdata.bin', 'rb') as f:
    data = f.read()

print_mem()
print("del data")
del data

print_mem()

The results are:

----------
ru_maxrss: 22.89MiB
virtual_memory.used: 8125.66MiB
allocating large array (80e6,)...
----------
ru_maxrss: 633.20MiB
virtual_memory.used: 8731.85MiB
del a
----------
ru_maxrss: 633.20MiB
virtual_memory.used: 8121.66MiB
read testdata.bin (~400MiB)
----------
ru_maxrss: 633.20MiB
virtual_memory.used: 8513.11MiB
del data
----------
ru_maxrss: 633.20MiB
virtual_memory.used: 8123.22MiB

It is clear how the ru_maxrss remembers the maximum RSS, but the current usage has dropped in the end.

Note on psutil.virtual_memory().used:

used: memory used, calculated differently depending on the platform and designed for informational purposes only.

Community
  • 1
  • 1
Ilja Everilä
  • 50,538
  • 7
  • 126
  • 127
  • But htop, top and 'cat /proc//stat' is showing the same 14 MB RES. Are htop and top also showing the max RES as well and not the current usage? Also should not we be worried about the RAM usage rather than the virtual memory used? – Shinto C V Apr 15 '16 at 10:11
  • In that our systems differ. Using your script with an added `raw_input()` right after `content = f.read()` to stop progression for observing RSS I see it raise to 10MiB for the time of the function `fileopen`, after which it drops back to 6.7MiB. Virtual memory usage can be a problem for 32bit systems, if it accumulates per process. – Ilja Everilä Apr 15 '16 at 10:19
  • Also an interesting question is: does the RSS raise, if you repeat `fun(sys.argv[1])` in a (infinite) loop? Again on this machine the answer is no, it's a stable 10.8MiB, but this time when I break the loop, the memory is not reclaimed by the OS and I see the same effect of RSS staying at 10.8MiB. Perhaps the OS decided that since it seems the program constantly needs that memory, why reclaim it in between. – Ilja Everilä Apr 15 '16 at 10:28
  • It seems that even calling `fun(...)` twice in a row is enough to trigger the effect. – Ilja Everilä Apr 15 '16 at 10:33
  • Which OS do you use? On Windows it does return to OS. On linux it does not. I tried it calling twice and yes the memory does not grow. It stays there. I would prefer if it reduce after the use. But that is not happening. – Shinto C V Apr 15 '16 at 11:12
  • Linux localhost 4.2.0-27-generic #32-Ubuntu SMP Fri Jan 22 04:49:08 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux – Ilja Everilä Apr 15 '16 at 11:16
  • Try reading results of searching ["linux return memory to os"](http://stackoverflow.com/search?q=linux+return+memory+to+os), there are many good answers that explain the details of when and why free'd memory is or isn't returned. – Ilja Everilä Apr 15 '16 at 11:19
  • Also what do you think about this discussion http://stackoverflow.com/questions/22768559/django-celery-memory-not-release ? – Shinto C V Apr 15 '16 at 11:19
  • Seems spot on. A true leak is surely a problem :) – Ilja Everilä Apr 15 '16 at 11:24
  • Btw the observed "2 calls to `fun` cause memory not to release" could also be a consequence of how mallopt(3) M_MMAP_THRESHOLD behaves in a modern linux: it is dynamic. The 1. allocation might get memory from mmap, the second from moving program break with sbrk. Etc etc... – Ilja Everilä Apr 15 '16 at 11:58