I have multiple pickle files which contain two lists. The first one is a lightweight list of floats, but the second one is a list of long vectors which consumes a lot of memory when loaded. How can I load only the first list from each pickle file? This is the way I currently do it:
import cPickle, bz2
with bz2.BZ2File(lhl_file, "rb") as pickled_file:
pickle_object = cPickle.load(pickled_file)
light_list, \
heavy_list = list(pickle_object)
If you pickled two lists individually into a pickle file, you should be able to load them individually easily.
Here is an example I adapted from another answer, which covers more thoroughly.
import pickle
filename = "pickle.dat"
data1 = 'hello'
data2 = 'world'
data3 = 'buddy'
# pickle one object at at time
f1 = open(filename, "wb")
pickle.dump(data1, f1)
pickle.dump(data2, f1)
pickle.dump(data3, f1)
f1.close()
# so that you can load them one at at time
f2 = open(filename, 'rb')
print pickle.load(f2)
print pickle.load(f2)
f2.close()
Output:
hello
world
PS: As far as I know, it's not possible to load data2 before data1. Though that is not the case for you it seems.
