pickle - putting more than 1 object in a file?

Question

I have got a method which dumps a number of pickled objects (tuples, actually) into a file.

I do not want to put them into one list, I really want to dump several times into the same file. My problem is, how do I load the objects again? The first and second object are just one line long, so this works with readlines. But all the others are longer. naturally, if I try

myob = cpickle.load(g1.readlines()[2])

where g1 is the file, I get an EOF error because my pickled object is longer than one line. Is there a way to get just my pickled object?

Why is it a problem to put them all in one list? – Russell Borogove Mar 17 '13 at 17:24 — Russell Borogove, Mar 17 '13 at 17:24

score 81 · Answer 1 · edited Oct 10 '13 at 21:04

81

If you pass the filehandle directly into pickle you can get the result you want.

import pickle

# write a file
f = open("example", "w")
pickle.dump(["hello", "world"], f)
pickle.dump([2, 3], f)
f.close()

f = open("example", "r")
value1 = pickle.load(f)
value2 = pickle.load(f)
f.close()

pickle.dump will append to the end of the file, so you can call it multiple times to write multiple values.

pickle.load will read only enough from the file to get the first value, leaving the filehandle open and pointed at the start of the next object in the file. The second call will then read the second object, and leave the file pointer at the end of the file. A third call will fail with an EOFError as you'd expect.

Although I used plain old pickle in my example, this technique works just the same with cPickle.

edited Oct 10 '13 at 21:04

scdwyer

3
2

answered Mar 17 '13 at 16:55

Martin Atkins

62,420
8
120
138

Also see the answer for reading multiple pickles from this answer. [Saving and loading multiple objects in python pickle file](http://stackoverflow.com/questions/20716812/saving-and-loading-multiple-objects-in-python-pickle-file) – salomonvh Sep 28 '15 at 12:17
does this also work with cpickle? (apparently not...) – dreamflasher Feb 27 '17 at 09:15

Samuel · Answer 2 · 2014-04-14T09:28:08.973

23

I think the best way is to pack your data into a single object before you store it, and unpack it after loading it. Here's an example using a tuple as the container(you can use dict also):

a = [1,2]
b = [3,4]

with open("tmp.pickle", "wb") as f:
    pickle.dump((a,b), f)

with open("tmp.pickle", "rb") as f:
    a,b = pickle.load(f)

edited Apr 14 '14 at 09:28

answered Apr 14 '14 at 09:14

Samuel

5,977
14
55
77

4

The op specifically said 'I do not want to put all the objects into one list.' He wants to write them sequentially. – Nick Crews Dec 26 '16 at 16:30
@Nick: That may be true, however the OP didn't say they didn't want to put them in a `tuple`. – martineau Jan 16 '18 at 00:37

score 2 · Answer 3 · edited May 23 '17 at 11:47

2

Don't try reading them back as lines of the file, justpickle.load()the number of objects you want. See my answer to the question How to save an object in Python for an example of doing that.

edited May 23 '17 at 11:47

Community

1
1

answered Mar 17 '13 at 16:54

martineau

119,623
25
170
301

pickle - putting more than 1 object in a file?

3 Answers3

Linked