2

I'm working on a project where I temporarily store objects in lists. But it seems that even when all references to these objects are deleted, the memory isn't freed, which leads to very high RAM usage in my process.

I've tried to reproduce a simple example and the results are indeed very strange, I can't explain them.

Here's a simple bench.py

import psutil
import os
import gc
import sys
import numpy as np

mod = int(sys.argv[1])
process = psutil.Process(os.getpid())
print(f"Empty memory: {process.memory_info().rss/1e6:.1f}MB")

arrs = list()
for i in range(1000):
    arr = np.arange(50000) * i
    if i % mod == 0:
        arrs.append(arr)
print(f"Memory before cleaning: {process.memory_info().rss/1e6:.1f}MB - {len(arrs)} elements in array")
del arrs
gc.collect()
print(f"Memory after cleaning: {process.memory_info().rss/1e6:.1f}MB")

If I run it with 1, 2, 3 and 500 arguments I get these results.

# 1
Empty memory: 33.3MB
Memory before cleaning: 435.4MB - 1000 elements in array
Memory after cleaning: 34.4MB

#2
Empty memory: 33.3MB
Memory before cleaning: 234.2MB - 500 elements in array
Memory after cleaning: 34.5MB

#3 - Why nothing is cleaned in that case ?
Empty memory: 33.3MB
Memory before cleaning: 167.8MB - 334 elements in array
Memory after cleaning: 167.8MB

#500
Empty memory: 33.3MB
Memory before cleaning: 35.4MB - 2 elements in array
Memory after cleaning: 34.4MB

It doesn't make any sense to me, why is the memory not cleaned the same way in the 3 cases ? Why the biggest array get the most efficient cleaning ?

RobinFrcd
  • 4,439
  • 4
  • 25
  • 49
  • 1
    while the objects are freed, that doesn't mean your process will necessarily release the memory back to the OS (allthough, it will be free for your process to use!)... Forr example, lots of small objects can cause fragmentation which not allow the process to release all the memory back to the OS (see: https://stackoverflow.com/questions/3770457/what-is-memory-fragmentation) – juanpa.arrivillaga Apr 04 '23 at 22:03
  • 1
    btw, terminology note, these are **lists** – juanpa.arrivillaga Apr 04 '23 at 22:06
  • 1
    And `gc.collect()` shouldn't be necessary here, since `gc` only controls the *cyclic garbage collector*, but there shouldn't be any reference cycles created in your code – juanpa.arrivillaga Apr 04 '23 at 22:06
  • Also note, you *can't "delete" objects in Python*. Python is a fully memory-managed language. `del` does *not* delete objects, it deletes *a name from a namespace* – juanpa.arrivillaga Apr 04 '23 at 22:20
  • Thanks! I said objects because in the real usecase, they are (images in a numpy format). So there's no way to force python to freeup the memory it allocated for these objects ? Because I'm kinda stuck with my process taking more than 5GB of RAM :/ – RobinFrcd Apr 04 '23 at 22:26
  • There is no way to manually manage memory in Python. Depending on what you are doing, you could create a subprocess (which has some overhead) which will (on most operating systems) free up the memory when the process finishes. – juanpa.arrivillaga Apr 04 '23 at 22:27
  • So, see the answers here: https://stackoverflow.com/questions/15455048/releasing-memory-in-python – juanpa.arrivillaga Apr 04 '23 at 22:27
  • @RobinFrcd unlike your list example, numpy frees the memory to the OS the moment you no-longer have references to the object, you most likely have a reference to them somewhere. – Ahmed AEK Apr 04 '23 at 22:40
  • @AhmedAEK I guess you mean it explicitly `free`'s the underlying buffer? Any handy link that elaborates on this? – juanpa.arrivillaga Apr 04 '23 at 22:42
  • 1
    @juanpa.arrivillaga something like this answer, the default memory allocator is still the same till now, https://stackoverflow.com/a/55001912/15649230 – Ahmed AEK Apr 05 '23 at 00:30
  • I've edited the snippet with NumPy arrays instead of simple list, the issue is the same. For example, in the case where I create a list with 1500 arrays in it, the process still uses 400+MB after cleanup. So for sure, numpy doesn't free all the memory. – RobinFrcd Apr 05 '23 at 09:05
  • Well, i can't replicate this on windows so i guess that's up to the OS allocator to not free that memory in the hope that it will reuse it later .... Or you need to update python or your glibc, But rest assured that python did free it back to the OS and there is no problem in the code – Ahmed AEK Apr 05 '23 at 10:41
  • What operating system are you using? – Kelly Bundy Apr 05 '23 at 10:54
  • I'm on Linux (`Linux pop-os 6.2.6-76060206-generic`), I've tried with both Python `3.9.13` and `3.11.2`. Same issue everywhere :/ – RobinFrcd Apr 05 '23 at 11:00

0 Answers0