Python - why does time.sleep cause memory leak?

Question

When I ran the code below, the memory was increasing. However if I deleted time.sleep(3), it was 0.1 in top and never increased.

It seems process not be terminated correctly, but why?

Code(Python 2.7.11):

import time
import multiprocessing

def process():
    #: FIXME
    time.sleep(3)
    return

def main():
    pool = multiprocessing.Pool(processes=10)
    while 1:
        pool.apply_async(process)
    pool.close()
    pool.join()

if __name__ == '__main__':
    main()

@vks the official document said `One must call close() or terminate() before using join()` https://docs.python.org/2/library/multiprocessing.html#multiprocessing.pool.multiprocessing.Pool.join — tanglong, Jan 19 '16 at 10:26
The following question might be interesting for you:http://stackoverflow.com/questions/7648967/python-multiprocessing-how-to-release-memory-when-a-process-is-done Check the answer by user1914881: "Try setting the maxtasksperchild argument on the pool. If you don't, then the process is reusued over and over again by the pool so the memory is never released. When set, the process will be allowed to die and a new one created in it's place. That will effectively clean up the memory." The processes seem to hold a bit of memory which is never released in your while 1 loop. (Though I am not sure) — Robin Spiess, Jan 19 '16 at 10:32

score 1 · Accepted Answer · answered Jan 19 '16 at 12:13

As far as I know, I think that as you spawn new processes in the same Pool, the garbage collection is never done, so you don't release memory from old processes, even if you are done using them. One fix would be to enforce the garbage collection in your while loop:

import time
import multiprocessing
import gc

def process():
    time.sleep(3)
    return

def main():
    pool = multiprocessing.Pool(processes=10)
    while 1:
        pool.apply_async(process)
        gc.collect()
    pool.close()
    pool.join()


if __name__ == '__main__':
    main()

This fixed the memory leaks for me, as you force the garbage collection before launching another set of processes. I hope someone could explain in a more detailed way the reason behing this memory leak.

Python - why does time.sleep cause memory leak?

1 Answers1

Linked