1

I have some Pickle objects stored in HDF5. When I load and then subsequently delete them, they free some arbitrarily small amount of memory and refuse to free the rest.

Here is some code:

import ctypes
import pickle
from memory_profiler import profile
import h5py
import hdf5plugin
import gc

f = h5py.File("mytestfile2.hdf5", "r")

@profile
def load():
    models = []
    for i in range(10):
        models.append(pickle.loads(f["model0"][0]))
    del models
    gc.collect()
    ctypes.CDLL("libc.so.6").malloc_trim(0)

When the model list is deleted, some memory is freed but even calling gc.collect() doesn't clear the rest. Calling malloc_trim does. This however doesn't seem super safe/clean to me? What would you use in prod? I will constantly be loading in gigabytes of pickles and clearing them once done with them.

Line #    Mem usage    Increment  Occurences   Line Contents
============================================================
    16    114.2 MiB    114.2 MiB           1   @profile
    17                                         def load():
    18    114.2 MiB      0.0 MiB           1       models = []
    19   8768.3 MiB      0.0 MiB          11       for i in range(10):
    20   8768.3 MiB   8654.1 MiB          10           models.append(pickle.loads(f["model0"][0]))
    21   6918.6 MiB  -1849.6 MiB           1       del models
    22   6918.6 MiB      0.0 MiB           1       gc.collect()
    23    115.5 MiB  -6803.1 MiB           1       ctypes.CDLL("libc.so.6").malloc_trim(0)

The alternative is doing something similar to: How do i use subprocesses to force python to release memory?

Alexis Drakopoulos
  • 1,115
  • 7
  • 22

0 Answers0