How to recover a pickled class and its instances

Question

I would like to store a class and many instances for later use, or to give to someone else.

So far I can pickle and recover the instances, but I have to recreate the class by hand before loading them.

I've looked at this documentation which leads me to believe I should be able to do this somehow, but I can't seem to find out exactly how to do it.

EDIT: I've read this answer discussing the use of dill(see this answer also), but I don't have dill installed. I'd like a pickle solution if it exists.

import numpy as np
import pickle

class wow(object):
    def __init__(self, x):
        self.x = x

w5 = wow(np.arange(5))
w3 = wow(range(3))

with open("w5w3.pickle", "w") as outfile:
    pickle.dump([w5, w3], outfile)

# save the class also
with open("wow.pickle", "w") as outfile:
    pickle.dump(wow, outfile)

# OK, now delete class wow, then try to recover the pickles
del wow, w3, w5

try:
    with open("wow.pickle", "r") as infile:
        wow = pickle.load(infile)

except Exception, e:  # returns: "'module' object has no attribute 'wow'"
    print str(e)
    print "so manually recreate class wow"

    class wow(object):
        def __init__(self, x):
            self.x = x  

with open("w5w3.pickle", "r") as infile:
    W = pickle.load(infile)

for thing in W:
    print type(thing.x), thing.x

have you tried pickling the class too? `pickle.dump([wow, w5, w3], outfile)` — Adam Smith, Dec 14 '15 at 07:08
i think you have to use 'wb' and 'rb' to binary writing and reading files — Netwave, Dec 14 '15 at 08:22
`dill` installs with `pip install`… and is pure python… so why don't you want to install it? Of course, being the `dill` author I'm biased… but with `dill`, you have a simple solution to your problem. — Mike McKerns, Dec 16 '15 at 02:58
I learned much more by asking this question and then understanding why specifically pickle will not pickle a class. There are also security issues with `dill` that I don't yet completely understand. Reflexively installing anything that some stranger tells you to install is not always wise. I think the best solution is the one in the accepted answer: send the class as readable python, and the objects in a pickle. — uhoh, Dec 16 '15 at 10:44
**NOTE:** the security issue(s) applies similarly to pickle and dill. Both might contain some naughty python (which you can't easily see afterward) as well as maliciously inserted instructions after pickling - see all three answers to this [question](http://stackoverflow.com/q/34314289/3904031) — uhoh, Dec 16 '15 at 15:21

score 5 · Accepted Answer · edited Jun 20 '20 at 09:12

I believe the error is caused because you deleted the class definition. Object serialization in Python (which to my knowledge is also in Java) requires the class definition to be there.

From your linked documentation:

Note that functions (built-in and user-defined) are pickled by “fully qualified” name reference, not by value. This means that only the function name is pickled, along with the name of the module the function is defined in. Neither the function’s code, nor any of its function attributes are pickled. Thus the defining module must be importable in the unpickling environment, and the module must contain the named object, otherwise an exception will be raised. [4]

Similarly, classes are pickled by named reference, so the same restrictions in the unpickling environment apply. Note that none of the class’s code or data is pickled

If you want to send your friend the class and instances, send the class through a code defining the class wow, and the instances through the pickle file.

OK that seems to be cut and dry, thanks! I'm still not comfortable with the comment I made on the answer from @LaPriWa "Section 11.1.4 "The following types can be pickled: ... classes that are defined at the top level of a module" which seems to me to say I should be able to do this" but I guess taken together - you can pickle the class but you won't get what you want back. — uhoh, Dec 14 '15 at 09:00
You're welcome. As for that line, yeah, it seems to be confusing, I guess you can pickle the class to serialize any runtime changes to the class variables to be retrieved later, for example. EDIT: sorry, I just remember that the class data is also not pickled. Not sure what we could gain by pickling a class, then — justhalf, Dec 14 '15 at 09:02

score 1 · Answer 2 · answered Dec 14 '15 at 08:44

1

Generally you can pickle any object if you can pickle every attribute of that object. Classes, functions, and methods cannot be pickled.

Source: https://wiki.python.org/moin/UsingPickle

Don't destroy the class or import it as a module.

answered Dec 14 '15 at 08:44

LaPriWa

1,787
1
12
19

That's a definitive answer, but I'm trying to see how that meshes with bullet #7 in [this](https://docs.python.org/2/library/pickle.html#what-can-be-pickled-and-unpickled) documentation. Section 11.1.4 "The following types can be pickled: ... classes that are defined at the top level of a module" which seems to me to say I should be able to do this. – uhoh Dec 14 '15 at 08:52
I guess it should say "you can pickle your Class but you can't get your class from a pickle." – uhoh Dec 14 '15 at 09:03

score 1 · Answer 3 · edited May 23 '17 at 10:27

If you are concerned about security, both dill and pickle have the same issues. However, if you want ease of use, dill handles the exact case you are looking to handle. You can dynamically create a class and then save it's instance with dill… then pass the instance to an environment that doesn't have the class definition, and dill will still be able to recreate the class and the class instance. This is because, unlike pickle, dill stores the class definition in the pickle (by default) as well as the class instance's state.

>>> class Foo(object):
...   x = 1
...   def bar(self, y):
...     return self.x + y
...   def __init__(self, z):
...     self.z = z
... 
>>> f = Foo(2)
>>> import dill
>>> with open('instance.pkl', 'w') as pkl: 
...   dill.dump(f, pkl)
... 
>>>

Then ship the file to another computer… restart a session, and like magic, it works.

Python 2.7.10 (default, Sep  2 2015, 17:36:25) 
[GCC 4.2.1 Compatible Apple LLVM 5.1 (clang-503.0.40)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import dill
>>> with open('instance.pkl', 'r') as pkl:
...   f = dill.load(pkl)
... 
>>> f
<__main__.Foo object at 0x10eb4a450>
>>> f.x
1
>>> f.z
2
>>> print dill.source.getsource(f.__class__)
class Foo(object):
  x = 1
  def bar(self, y):
    return self.x + y
  def __init__(self, z):
    self.z = z

>>>

In short, I disagree with all of the other answers. You've even linked an answer in your question that shows that class instances, class instance state, and class code can be pickled. Pickle python class instance plus definition

If you want to do the same thing using `pickle` and not `dill`, then you can. All you have to do is use `copyreg` to register how to serialize the class by saving the code. This is what `dill` does on top of `pickle`, so you can also do it by registering a definition that serializes a class. — Mike McKerns, Dec 21 '15 at 13:44
Thanks. The question is truly about pickle; pickle being that module which is imported when I type `import pickle`. — uhoh, Dec 22 '15 at 15:40

score -1 · Answer 4 · answered Dec 14 '15 at 08:40

-1

you must open pickle file in binary mode as it mentioned here:

with open('data.pickle', 'wb') as f:
    pickle.dump(data, f)

answered Dec 14 '15 at 08:40

Elika Mirzadeh

29
3

For the objects, it works fine as written `...'w')`. For the class, `...'wb')` and `...'rb')` do not change the error message `'module' object has no attribute 'wow'`. – uhoh Dec 14 '15 at 08:44

How to recover a pickled class and its instances

4 Answers4

Linked