Is this the right way to pickle instance methods? If yes, why isn't it in Python 3?

Question

Instance methods can not automatically be pickled in both Python 2 or Python 3.

I need to pickle instance methods with Python 3 and I ported example code of Steven Bethard to Python 3:

import copyreg
import types

def _pickle_method(method):
    func_name = method.__func__.__name__
    obj = method.__self__
    cls = method.__self__.__class__
    return _unpickle_method, (func_name, obj, cls)

def _unpickle_method(func_name, obj, cls):
    for cls in cls.mro():
        try:
            func = cls.__dict__[func_name]
        except KeyError:
            pass
        else:
            break
    return func.__get__(obj, cls)

copyreg.pickle(types.MethodType, _pickle_method, _unpickle_method)

Is this method fool proof for pickling instance methods? Or can some things go horribly wrong? I have tested it with some mock up classes and everything seem to work.

If nothing can go wrong, why isn't it possible in Python 3 to standard pickle instance methods?

Because all references must be globally accessible. http://www.velocityreviews.com/forums/t395502-why-cant-you-pickle-instancemethods.html — Lennart Regebro, Jul 05 '11 at 20:48
@Lennart If you look to the link, you'll see that I found the same thread myself. It however does not very well explain what the real difference would be between pickling a method and function. I know that also some functions, like a lambda function can not be pickled, but in that case an error is thrown when it is not possible. Why not the same for methods? — Peter Smit, Jul 05 '11 at 20:54

score 0 · Answer 1 · answered Sep 24 '15 at 12:21

I can't actually reproduce the original issue on python 3.5.0, see the following example. It might be worth checking on the latest version to see if it just works out-of-the-box :-)

import pickle
import sys

class Foo:

    @staticmethod
    def my_func():
        return 'Foo.my_func'


class Bar:
    pass

if __name__ == '__main__':
    if len(sys.argv) > 1 and sys.argv[1] == 'restore':
        print('loading pickle')
        with open('test.pickle', 'rb') as fp:
            restored = pickle.load(fp)
            print(restored.baz())
    else:
        print('saving pickle')
        b = Bar()
        b.baz = Foo.my_func

        with open('test.pickle', 'w+b') as fp:
            p = pickle.dump(b, fp)

(test)~/test$ python test.py

saving pickle

(test)~/test$ python test.py restore

loading pickle

Foo.my_func

score 0 · Answer 2 · edited May 23 '17 at 11:45

If you want to pickle class instances (and instance methods), just use dill...

>>> import dill
>>> 
>>> class Foo:
...   def bar(self, x):
...     return self.y + x
...   def zap(self, y):
...     self.y = y
...   y = 1
... 
>>> f = Foo()
>>> f.zap(4)
>>> f.monty = 'python'
>>> 
>>> _f = dill.dumps(f)
>>> del Foo
>>> del f
>>> f = dill.loads(_f)
>>> f.monty
'python'
>>> f.y
4
>>> 
>>> _b = dill.dumps(f.bar)
>>> del f
>>> bar = dill.loads(_b)
>>> bar(4)
8
>>>

dill works when you delete the class object, as seen above… so it also works if you start a fresh python session without the class defined, or if you change the class definition. dill even works when you don't have an instance of the class object, or a class instance available. If you want to see how to do it, look at the dill source code: https://github.com/uqfoundation/dill

Also see related: https://stackoverflow.com/a/21345273/2379433

Is this the right way to pickle instance methods? If yes, why isn't it in Python 3?

2 Answers2

Linked