3

Here is a python program I'm running with numpy. Why does it seems to use so much memory even after I delete the np.array? Is this normal? Or Is there something I'm not understanding about memory and how it is allocated?

Python 3.7.9 (default, Aug 31 2020, 12:42:55)
[GCC 7.3.0] :: Anaconda, Inc. on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import os, psutil
>>> def memory():
...     pid = os.getpid()
...     proc = psutil.Process(pid)
...     mem = proc.memory_info()
...     print('Memory used: {:.2f} MB'.format(mem.vms/1024/1024))
...
>>> import numpy as np
>>> memory()
Memory used: 269.27 MB
>>> a = np.random.rand(100,100)
>>> memory()
Memory used: 269.27 MB
>>> b = a @ a
>>> memory()
Memory used: 2932.79 MB
>>> del b
>>> memory()
Memory used: 2932.79 MB
Unskilled
  • 131
  • 2
  • try import gc; gc.collect() – Stefano Borini Nov 19 '20 at 23:59
  • this didn't seem to make any difference – Unskilled Nov 20 '20 at 00:03
  • Pretty much a duplicte: https://stackoverflow.com/questions/18310668/is-freeing-handled-differently-for-small-large-numpy-arrays – juanpa.arrivillaga Nov 20 '20 at 00:27
  • In general, you can make sure that the array is freed by making sure you don't keep errant references around, what you *don't* control is if that memory is released back to the OS, from the OS POV, the memory is allocated, although in your process, it's free to be re-used. – juanpa.arrivillaga Nov 20 '20 at 00:28
  • @Unskilled you wouldn't predict `gc.collect()` to make a difference. `gc` controls the *auxilliary cyclic garbage collector*, which handles reference cycles. The main memory-management strategy in CPython is reference counting, and `gc` is usually not relevant unless you are creating reference cycles (which you aren't) – juanpa.arrivillaga Nov 20 '20 at 00:29
  • @juanpa.arrivillaga yes and no. If I remember correctly, even if something is reference counted, the arena does not necessarily release the memory. The memory management layer in python has changed quite a lot in version 3 – Stefano Borini Nov 20 '20 at 15:29

0 Answers0