I have some Pickle objects stored in HDF5. When I load and then subsequently delete them, they free some arbitrarily small amount of memory and refuse to free the rest.
Here is some code:
import ctypes
import pickle
from memory_profiler import profile
import h5py
import hdf5plugin
import gc
f = h5py.File("mytestfile2.hdf5", "r")
@profile
def load():
models = []
for i in range(10):
models.append(pickle.loads(f["model0"][0]))
del models
gc.collect()
ctypes.CDLL("libc.so.6").malloc_trim(0)
When the model list is deleted, some memory is freed but even calling gc.collect() doesn't clear the rest. Calling malloc_trim does. This however doesn't seem super safe/clean to me? What would you use in prod? I will constantly be loading in gigabytes of pickles and clearing them once done with them.
Line # Mem usage Increment Occurences Line Contents
============================================================
16 114.2 MiB 114.2 MiB 1 @profile
17 def load():
18 114.2 MiB 0.0 MiB 1 models = []
19 8768.3 MiB 0.0 MiB 11 for i in range(10):
20 8768.3 MiB 8654.1 MiB 10 models.append(pickle.loads(f["model0"][0]))
21 6918.6 MiB -1849.6 MiB 1 del models
22 6918.6 MiB 0.0 MiB 1 gc.collect()
23 115.5 MiB -6803.1 MiB 1 ctypes.CDLL("libc.so.6").malloc_trim(0)
The alternative is doing something similar to: How do i use subprocesses to force python to release memory?