13

In my case, I wish to pickle (using pickle.dump()) two separate lists to a file, then retrieve these from a separate file, however when using pickle.load() I have struggled finding where one list ends and the next begins as I simply don't know how to pickle.dump() them in a manner that makes them easy to retrieve, even after looking through documentation.

MackM
  • 2,906
  • 5
  • 31
  • 45
Drake
  • 159
  • 1
  • 1
  • 8
  • Have you tried writing them as sublists in an enclosing list e.g. `my_list = [[list_1], [list_2]])` or similar or in some way incorporate them into a single data structure to pickle? – roganjosh Feb 20 '17 at 17:21
  • If you pickle two lists one after another in the same file with `pickle.dump(l1,f); pickle.dump(l2,f)`, then you can unpickle them in the same order from the same file with `l1=pickle.load(f); l2=pickle.load(f)`. Pickle will find where one ends and the other starts. – DYZ Feb 20 '17 at 17:23
  • Ill try that out, thank you. If that doesn't work Ill use sublists, which I didn't think of. – Drake Feb 20 '17 at 17:27
  • Why not pickle a tuple of your lists? `pickle.dump((l1, l2), fh)` and `l1, l2 = pickle.load(fh)`? – Ben Feb 20 '17 at 17:32
  • Thank you I got it to work! Ill try using some of your other suggestions to make it as efficient as possible. – Drake Feb 20 '17 at 17:40

1 Answers1

23

pickle will read them in the same order you dumped them in.

import pickle

test1, test2 = ["One", "Two", "Three"], ["1", "2", "3"]
with open("C:/temp/test.pickle","wb") as f:
    pickle.dump(test1, f)
    pickle.dump(test2, f)
with open("C:/temp/test.pickle", "rb") as f:
    testout1 = pickle.load(f)
    testout2 = pickle.load(f)

print testout1, testout2

Prints out ['One', 'Two', 'Three'] ['1', '2', '3']. To pickle an arbitrary number of objects, or to just make them easier to work with, you can put them in a tuple, and then you only have to pickle the one object.

import pickle

test1, test2 = ["One", "Two", "Three"], ["1", "2", "3"]
saveObject = (test1, test2)
with open("C:/temp/test.pickle","wb") as f:
    pickle.dump(saveObject, f)
with open("C:/temp/test.pickle", "rb") as f:
    testout = pickle.load(f)

print testout[0], testout[1]

Prints out ['One', 'Two', 'Three'] ['1', '2', '3']

MackM
  • 2,906
  • 5
  • 31
  • 45
  • Thank you, I now know pickle reads them in the order they were pickled. – Drake Feb 20 '17 at 20:24
  • 1
    a tip for people landing here wanting to save multiple dataframes into one pickle. You can similarly to the above, but saving all the dataframes into a dictionary: `df_dicts = { }` then `df_dicts[0], df_dicts[1], df_dicts[2] = df1, df2, df3`. When dumping, use just `df_dicts`. – mrbTT Sep 20 '18 at 21:40
  • Should `with open..."wb"` be followed by an `f.close()`? – Dan Sep 24 '19 at 15:33
  • 1
    @Dan [`with`](https://www.geeksforgeeks.org/with-statement-in-python/) takes care of it for you! – MackM Sep 24 '19 at 20:00