I am coming from this answer: https://stackoverflow.com/a/66971031/3864842
Essentially, I am having a memory leak using tf.data and the solution that seems to work for many people, is to use libtcmalloc_minimal.so.4 instead of the system's memory allocator.
I have installed the library and used it like this:
sudo apt-get install libtcmalloc-minimal4
LD_PRELOAD="/lib/x86_64-linux-gnu/libtcmalloc_minimal.so.4" python main.py ...
The first run right after the installation worked flawlessly. But after restarting the script, I experienced the leaks again. (same shell, Same script, same command)
Am I doing something wrong? Can I somehow check if Tensorflow/Python is actually using the library?
So far it worked for less than 10% of all test runs (about 50).
