How to free memory of python deleted object?

Question

It seems python3.5 does not completely free memory of any deleted object, this may because of python internally maintaining some kind of memroy pool for reusing purpose, however, I don't want to reuse them, and I want to free them to make memory available for other programs running on linux.

>>> psutil.Process().memory_info().rss / 2**20
11.47265625
>>> d = {x:x for x in range(10**7)}
>>> psutil.Process().memory_info().rss / 2**20
897.1796875
>>> del d
>>> gc.collect()
0
>>> psutil.Process().memory_info().rss / 2**20
15.5859375

This is just a toy example, the real problem is on a running server, taking 20GB of unfreeable memory.

here is another example: (wd1 is a dict of dict with string keys)

>>> psutil.Process().memory_info().rss / 2**20
28.1796875
>>> wd1 = {x:{i:i for i in d} for x in k}
>>> psutil.Process().memory_info().rss / 2**20
682.78125
>>> del wd1
>>> psutil.Process().memory_info().rss / 2**20
186.21484375

You have no control over memory management in python (or any other modern language with automatic or semi automatic memory management). — Daniel, Jul 23 '16 at 19:23
This is really a problem when we need to use python on a server program, how to modify python3.5's source code to make it usable on servers? — V Y, Jul 23 '16 at 20:22
Just a thought, but what about using subprocess or the multiprocess modules to do the big chunks of work? Once a Linux process terminates, wouldn't you get all your memory back? *how to modify python3.5's source...* seems like a no-starter, to me, at least. — JL Peyret, Jul 23 '16 at 20:39
oh, and you may also want to look at *weakref* and see if any of it applies to your issues. — JL Peyret, Jul 23 '16 at 20:51
20GB is way to much for some maintaining memory. You should show the code where you get "unfreeable" memory. You probably still keep references to all of your objects. — Daniel, Jul 23 '16 at 22:00
@Daniel I can't post production code here, but the sample code is very similar, except "d" is a 1000x larger dict, and being updated inside a function. — V Y, Jul 23 '16 at 22:53
@JLPeyret I can't use child process, because multiprocessing does not have a shared dict — V Y, Jul 23 '16 at 22:57
If you repeat those instructions *again*, what is the result? I.e. if you repeat those steps multiple times in the same python process, does the return value of `psutil.Process().memory_info().rss` level off, or does it keep growing? — Warren Weckesser, Jul 23 '16 at 23:17
@WarrenWeckesser on python interpreter, it grows a little but not much, on server, it grows a lot by repeating it. — V Y, Jul 24 '16 at 00:03
@JLPeyret d = weakref.WeakKeyDictionary({x:x for x in range(10**7)}) get a "TypeError: cannot create weak reference to 'int' object ", if it WeakKeyDictionary can not be used on 'int' or 'str' , I don't know how to use it — V Y, Jul 24 '16 at 00:07
Possible duplicate of [How can I explicitly free memory in Python?](https://stackoverflow.com/q/1316767/608639) — jww, Jan 20 '19 at 07:58

Steve Barnes · Answer 1 · 2016-07-24T04:45:09.393

Once you delete an object it is available to garbage collected rather than deleted immediately - so just give it some time and it will free up or trigger a gc.collect() to speed things up.

python.exe
Python 3.5.1 (v3.5.1:37a07cee5969, Dec  6 2015, 01:38:48) [MSC v.1900 32 bit (Intel)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> import psutil
>>> import gc
>>> psutil.Process().memory_info().rss / 2**20
13.2890625
>>> d = {x:x for x in range(10**7)}
>>> psutil.Process().memory_info().rss / 2**20
359.13671875
>>> del d
>>> psutil.Process().memory_info().rss / 2**20
13.5234375
>>> gc.collect()
0
>>> psutil.Process().memory_info().rss / 2**20
13.4375
>>>

Just for reference the Python 3 shell is actually more like ipython 2 in that there is a certain amount of storage taken up with history, etc., just for reference:

Python 3.5.1 (v3.5.1:37a07cee5969, Dec  6 2015, 01:38:48) [MSC v.1900 32 bit (Intel)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> import psutil
>>> psutil.Process().memory_info().rss / 2**20
13.1875
>>> psutil.Process().memory_info().rss / 2**20
13.20703125
>>> psutil.Process().memory_info().rss / 2**20
13.20703125
>>> psutil.Process().memory_info().rss / 2**20
13.20703125
>>> psutil.Process().memory_info().rss / 2**20
13.20703125
>>> 22*3
66
>>> psutil.Process().memory_info().rss / 2**20
13.25390625
>>> import gc
>>> psutil.Process().memory_info().rss / 2**20
13.25390625
>>> gc.collect()
0
>>> psutil.Process().memory_info().rss / 2**20
13.171875
>>>

Next Morning to check if doing dict update in a function is different:

>>> psutil.Process().memory_info().rss / 2**20
13.1484375
>>> D = {}
>>> psutil.Process().memory_info().rss / 2**20
13.1484375
>>> def UpdateD(d, v):
...     """ Add the text and value for v to dict d """
...     d[v] = str(v)
...
>>> psutil.Process().memory_info().rss / 2**20
13.16015625
>>> for x in range(10**7):
...     UpdateD(D, x)
...
>>> psutil.Process().memory_info().rss / 2**20
666.6328125
>>> del D
>>> psutil.Process().memory_info().rss / 2**20
10.765625
>>> gc.collect()
0
>>> psutil.Process().memory_info().rss / 2**20
12.8984375
>>>

So it looks like your production code might be hanging onto a reference that you still have to track down.

Just as a note for people not familiar with gc, it will temporarily block so definitely not something to call in the middle of time sensitive logic. — David, Jul 23 '16 at 19:13
I waited 30 minutes, and ran gc.collect() several times, it still shows 15.5859375, not freed — V Y, Jul 23 '16 at 19:18
the same thing happens in a server process, not just on python3 shell, unfreeable and not gc collectable — V Y, Jul 23 '16 at 20:28

How to free memory of python deleted object?

1 Answers1

Related