After unpickling system use a lot of RAM

Question

I have an object (defaultdict) with structure: { srting : [(string, (float, float)), (string, (float, float)), ....]}

The size of it is about 12,5 MB

I am pickling with code:

with open(Path_to_file, 'wb') as file:
    pickle.dump(data_dict, file)

Pickle file weights about 300 MB. In proccess unpickling with code:

with open(Path_to_file, 'rb') as file:
    data_dict_new = pickle.load(file)

system is using a lot of RAM (about 3,5 GB and more). But after unpickling Python uses about 1 GB of RAM.

So I have two questions:

1. What does keep in RAM apart of my structure?
2. How can I clean it?

gc.collect() doesn't help.

Answer 1

I was able to reproduce this. Indeed, if you're unpickling a large (about 300M) file, a lot of extra memory stay used. In my case, 1.6G was used by a process just to keep original generated data_dict, and 2.9G if I load it from file.

However, if you'll run unpickling in a subprocess, system will do a full memory clean after process join(). (as stated in this answer: https://stackoverflow.com/a/1316799/1102535). So the example of unpickling without extra memory used:

from multiprocessing import Process, Manager

def load_pickle(filename, data):
    import cPickle as pickle
    with open(filename, 'rb') as file:
        data_pkl = pickle.load(file)
    for key, val in data_pkl.iteritems():
        data[key] = val

manager = Manager()
data_dict = manager.dict()
p = Process(target=load_pickle, args=("test.pkl", data_dict))
p.start()
p.join()
print len(data_dict)

This code has its drawbacks (like copying between dicts), but at least you have the idea. As for me, it uses almost the same amount of memory after unpickling as for original data before pickling.