how to dereference a module and free memory in python?

Question

I am working on a project where I need to dereference a module and free its memory for further model training. Here I mentioned the demo code with memory uses after certain block. I used garbage collection as well as the del function but it couldn't worked for me.

import psutil
import sys
import gc
sys.path.insert(0,'/scripts_v2')

process = psutil.Process()
mem = process.memory_info().rss/(1024**2)
print(mem)

import pandas as pd

process = psutil.Process()
mem = process.memory_info().rss/(1024**2)
print(mem)

sys.modules.pop('pandas')
#del pd

gc.collect()

process = psutil.Process()
mem = process.memory_info().rss/(1024**2)
print(mem)

I calculated the memory after a specific block of code. Here I mentioned the output of the above code.

You can see that before and after deleting the pandas library, its memory is still 60.65 MB. How can I free its memory?

Does this answer your question? [How to de-import a Python module?](https://stackoverflow.com/questions/1668223/how-to-de-import-a-python-module) — XPhyro, Apr 02 '20 at 11:16
No. I am able to delete a module. I want to free its memory after deleting a module. As you can see from the attached image the memory used by the system is same before and after deleting the pandas library. — Sandeep Dahake, Apr 02 '20 at 11:24

XPhyro · Answer 1 · 2020-04-02T12:18:42.697

Deleting a variable, a module or anything, in Python, does not have to free its memory. Python simply caches some types when you initialise them.

Your problem is likely caused by the caching of some objects in the pandas library, and as far as I am aware, you cannot free the internal cache of Python. There might be a way to do it, but it will definitely be too hacky to even bother. If you need 100% control over your memory in some project then Python will not be helping you much. To get more information about the topic see the answers here and see especially this answer.

Note that del and gc.collect() might release memory, it is just that they do not have to.

Edit: sys.modules.pop('pandas') does not de-import pandas, you can see by doing perhaps print(pd.__version__). But, the same memory problem occurs if you actually de-import the library using the methods specified here.

how to dereference a module and free memory in python?

1 Answers1