143

I need to save about a dozen objects to a file and then restore them later. I've tried to use a for loop with pickle and shelve but it didn't work right.

Edit.
All of the objects that I was trying to save were in the same class (I should have mentioned this before), and I didn't realize that I could just save the whole class like this:

import pickle
def saveLoad(opt):
    global calc
    if opt == "save":
        f = file(filename, 'wb')
        pickle.dump(calc, f, 2)
        f.close
        print 'data saved'
    elif opt == "load":
        f = file(filename, 'rb')
        calc = pickle.load(f)
    else:
        print 'Invalid saveLoad option'
Cadoiz
  • 1,446
  • 21
  • 31
lunarfyre
  • 1,636
  • 2
  • 14
  • 17

7 Answers7

223

If you need to save multiple objects, you can simply put them in a single list, or tuple, for instance:

import pickle

# obj0, obj1, obj2 are created here...

# Saving the objects:
with open('objs.pkl', 'w') as f:  # Python 3: open(..., 'wb')
    pickle.dump([obj0, obj1, obj2], f)

# Getting back the objects:
with open('objs.pkl') as f:  # Python 3: open(..., 'rb')
    obj0, obj1, obj2 = pickle.load(f)

If you have a lot of data, you can reduce the file size by passing protocol=-1 to dump(); pickle will then use the best available protocol instead of the default historical (and more backward-compatible) protocol. In this case, the file must be opened in binary mode (wb and rb, respectively).

The binary mode should also be used with Python 3, as its default protocol produces binary (i.e. non-text) data (writing mode 'wb' and reading mode 'rb').

Eric O. Lebigot
  • 91,433
  • 48
  • 218
  • 260
  • 15
    In Python 3.5, I had to open the file in "byte" mode, e.g. `with open('objs.pickle', 'wb') as f:` (note the `wb`). – kbrose May 24 '16 at 23:17
  • 1
    Hi @Eric, what is the need of `with open('objs.pkl') as f:` compare to simply `obj1, obj2 = pickle.load(open("objs.pkl","rb"))` ? Is there any difference between these two? – mpx Jul 09 '20 at 15:50
  • 1
    With the second form you do not close the file. This is not considered good practice, as the number of files that you can be opened in parallel is usually quite limited by the operating systems (try a loop that opens files without closing them!). That said, in practice, not closing the file often works, when you don't open many files. – Eric O. Lebigot Jul 21 '20 at 14:33
59

There is a built-in library called pickle. Using pickle you can dump objects to a file and load them later.

import pickle

f = open('store.pckl', 'wb')
pickle.dump(obj, f)
f.close()

f = open('store.pckl', 'rb')
obj = pickle.load(f)
f.close()
Yossi
  • 11,778
  • 2
  • 53
  • 66
  • 1
    I Python 3.4 use: `f = open('store.pckl', 'wb')` to open a file to write to. Refer to http://stackoverflow.com/questions/13906623/using-pickle-dump-typeerror-must-be-str-not-bytes And use `f = open('store.pckl', 'rb') to open a file to read from. Refer to http://stackoverflow.com/questions/7031699/typeerror-str-does-not-support-the-buffer-interface. – user3731622 May 20 '15 at 22:53
  • is this specific to 3.4+? I almost voted the answer down because it generates errors when you don't use 'b'. – Wilmer E. Henao Jun 27 '16 at 20:22
  • 1
    The question was "How do I save and restore multiple variables in python?" Please update your answer how to handle multiple variables in a pickle, instead of one variable. – Ferro Feb 06 '21 at 21:17
14

Another approach to saving multiple variables to a pickle file is:

import pickle

a = 3; b = [11,223,435];
pickle.dump([a,b], open("trial.p", "wb"))

c,d = pickle.load(open("trial.p","rb"))

print(c,d) ## To verify
Guruprasad Raghavan
  • 149
  • 1
  • 3
  • This does not safely close the files. `open` makes a file handle that needs to be closed either by storing the handle and calling `.close` on it, or preferably by using `with open(...) as ...:`. The second option is generally better as it will close the file automatically for you, even if an exception occurs. – cdgraham Nov 06 '21 at 19:39
13

You should look at the shelve and pickle modules. If you need to store a lot of data it may be better to use a database

John La Rooy
  • 295,403
  • 53
  • 369
  • 502
  • I want to save a single object which logins into a cloud server, in order to handle if I login multiple times over time, the server rejects my request. Does dumping an object into a file using pickle module may have any security issue? , for example where if someone obtain my dumped object, than they can login into my cloud-storage without using a password. – alper Mar 15 '20 at 14:38
8

The following approach seems simple and can be used with variables of different size:

import hickle as hkl
# write variables to filename [a,b,c can be of any size]
hkl.dump([a,b,c], filename)

# load variables from filename
a,b,c = hkl.load(filename)
devil in the detail
  • 2,905
  • 17
  • 15
5

You could use klepto, which provides persistent caching to memory, disk, or database.

dude@hilbert>$ python
Python 2.7.6 (default, Nov 12 2013, 13:26:39) 
[GCC 4.2.1 Compatible Apple Clang 4.1 ((tags/Apple/clang-421.11.66))] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> from klepto.archives import file_archive
>>> db = file_archive('foo.txt')
>>> db['1'] = 1
>>> db['max'] = max
>>> squared = lambda x: x**2
>>> db['squared'] = squared
>>> def add(x,y):
...   return x+y
... 
>>> db['add'] = add
>>> class Foo(object):
...   y = 1
...   def bar(self, x):
...     return self.y + x
... 
>>> db['Foo'] = Foo
>>> f = Foo()
>>> db['f'] = f  
>>> db.dump()
>>>

Then, after interpreter restart...

dude@hilbert>$ python
Python 2.7.6 (default, Nov 12 2013, 13:26:39) 
[GCC 4.2.1 Compatible Apple Clang 4.1 ((tags/Apple/clang-421.11.66))] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> from klepto.archives import file_archive
>>> db = file_archive('foo.txt')
>>> db
file_archive('foo.txt', {}, cached=True)
>>> db.load()
>>> db
file_archive('foo.txt', {'1': 1, 'add': <function add at 0x10610a0c8>, 'f': <__main__.Foo object at 0x10510ced0>, 'max': <built-in function max>, 'Foo': <class '__main__.Foo'>, 'squared': <function <lambda> at 0x10610a1b8>}, cached=True)
>>> db['add'](2,3)
5
>>> db['squared'](3)
9
>>> db['f'].bar(4)
5
>>>

Get the code here: https://github.com/uqfoundation

Mike McKerns
  • 33,715
  • 8
  • 119
  • 139
0

Here is a solution using dill (https://dill.readthedocs.io/en/latest/dill.html) package. Pickle should work similarly.

""" Some objects to save """

import numpy as np
a = 6
b = 3.5
c = np.linspace(2,5,11)

""" Test the dump part """

import dill
file_name = 'dill_dump_test_file.dil'
list_of_variable_names = ('a', 'b', 'c')

with open(file_name,'wb') as file:
    dill.dump(list_of_variable_names, file)  # Store all the names first
    for variable_name in list_of_variable_names:
        dill.dump(eval(variable_name), file) # Store the objects themselves
        
#   Clear all the variables here     

""" Test the load part """

import dill
file_name = 'dill_dump_test_file.dil'
g = globals()

with open(file_name,'rb') as file:
    list_of_variable_names = dill.load(file)  # Get the names of stored objects
    for variable_name in list_of_variable_names:
        g[variable_name] = dill.load(file)    # Get the objects themselves
Cristian
  • 1
  • Your answer could be improved with additional supporting information. Please [edit] to add further details, such as citations or documentation, so that others can confirm that your answer is correct. You can find more information on how to write good answers [in the help center](/help/how-to-answer). – Community Jun 17 '23 at 01:57