2

dill is a great tool for pickling most the Python objects, I use it in IPython parallel to serialize calculations. One issue I keep getting into is around dill-ing class definitions. One of the errors I get is explained below.

While trying to serialize class definitions, I keep getting AssertionError from dill. I wonder why one of these works and the other fails:

class MyClassEmpty(object):
    pass

class MyClassInit(object):
    def __init__(self):
        super(MyClassInit).__init__()

dill.dumps(MyClassEmpty)  # returns: '\x80\x02cdill.dill\n_create_type\nq\x00(cdill.dill\n_load_type\nq\x01U\x08TypeTypeq\x02\x85q\x03Rq\x04U\x0cMyClassEmptyq\x05h\x01U\nObjectTypeq\x06\x85q\x07Rq\x08\x85q\t}q\n(U\n__module__q\x0bU\x08__main__q\x0cU\x07__doc__q\rNutq\x0eRq\x0f.'

dill.dumps(MyClassInit)  # AssertionError at line 244 of MyClassEmpty (assert id(obj) not in self.memo)

I'm on Python 2.7.6 using dill 0.2.2.

Mark Horvath
  • 1,136
  • 1
  • 9
  • 24
  • 1
    Have you read e.g. http://stackoverflow.com/questions/2626636/pickling-a-class-definition? Also, your `super` call seems incorrect - that should be `super(MyClassInit, self).__init__()`. – jonrsharpe Feb 16 '15 at 13:52
  • 1
    I can execute it, there's no AssertionError thrown. – SmCaterpillar Feb 16 '15 at 13:58
  • Thanks jonrsharpe! The issue was simply missing `, self`. Having corrected `dumps()` just works as expected, as in the linked post as well. – Mark Horvath Feb 16 '15 at 14:17
  • @SmCaterpillar, interesting! Are you on the same Python and dill version? – Mark Horvath Feb 16 '15 at 17:38
  • 1
    Hmm I used python 2.7.9, but within an IPython Notebook environment. Still that should not change anything?! – SmCaterpillar Feb 16 '15 at 17:53
  • Just updated the description, I'm on 2.7.6 which can make a difference. Have seen similar discrepancy with even minor changes in the interpreter. Thanks for noting! – Mark Horvath Feb 16 '15 at 18:34

1 Answers1

6

I'm the author of dill. The super issue should be resolved -- see: https://github.com/uqfoundation/dill/issues/26

>>> class MyClassEmpty(object):
...   pass
... 
>>> class MyClassInit(object):
...   def __init__(self):
...     super(MyClassInit).__init__()
... 
>>> import dill
>>> 
>>> dill.dumps(MyClassEmpty)
'\x80\x02cdill.dill\n_create_type\nq\x00(cdill.dill\n_load_type\nq\x01U\x08TypeTypeq\x02\x85q\x03Rq\x04U\x0cMyClassEmptyq\x05h\x01U\nObjectTypeq\x06\x85q\x07Rq\x08\x85q\t}q\n(U\n__module__q\x0bU\x08__main__q\x0cU\x07__doc__q\rNutq\x0eRq\x0f.'
>>> dill.dumps(MyClassInit) 
'\x80\x02cdill.dill\n_create_type\nq\x00(cdill.dill\n_load_type\nq\x01U\x08TypeTypeq\x02\x85q\x03Rq\x04U\x0bMyClassInitq\x05h\x01U\nObjectTypeq\x06\x85q\x07Rq\x08\x85q\t}q\n(U\n__module__q\x0bU\x08__main__q\x0cU\x07__doc__q\rNU\x08__init__q\x0ecdill.dill\n_create_function\nq\x0f(cdill.dill\n_unmarshal\nq\x10U\x90c\x01\x00\x00\x00\x01\x00\x00\x00\x02\x00\x00\x00C\x00\x00\x00s\x14\x00\x00\x00t\x00\x00t\x01\x00\x83\x01\x00j\x02\x00\x83\x00\x00\x01d\x00\x00S(\x01\x00\x00\x00N(\x03\x00\x00\x00t\x05\x00\x00\x00supert\x0b\x00\x00\x00MyClassInitt\x08\x00\x00\x00__init__(\x01\x00\x00\x00t\x04\x00\x00\x00self(\x00\x00\x00\x00(\x00\x00\x00\x00s\x07\x00\x00\x00<stdin>R\x02\x00\x00\x00\x02\x00\x00\x00s\x02\x00\x00\x00\x00\x01q\x11\x85q\x12Rq\x13c__builtin__\n__main__\nh\x0eNN}q\x14tq\x15Rq\x16utq\x17Rq\x18.'
>>> 
>>> class MyClassInit2(object):
...   def __init__(self):
...     super(MyClassInit, self).__init__()
... 
>>> dill.dumps(MyClassInit2)
'\x80\x02cdill.dill\n_create_type\nq\x00(cdill.dill\n_load_type\nq\x01U\x08TypeTypeq\x02\x85q\x03Rq\x04U\x0cMyClassInit2q\x05h\x01U\nObjectTypeq\x06\x85q\x07Rq\x08\x85q\t}q\n(U\n__module__q\x0bU\x08__main__q\x0cU\x07__doc__q\rNU\x08__init__q\x0ecdill.dill\n_create_function\nq\x0f(cdill.dill\n_unmarshal\nq\x10U\x93c\x01\x00\x00\x00\x01\x00\x00\x00\x03\x00\x00\x00C\x00\x00\x00s\x17\x00\x00\x00t\x00\x00t\x01\x00|\x00\x00\x83\x02\x00j\x02\x00\x83\x00\x00\x01d\x00\x00S(\x01\x00\x00\x00N(\x03\x00\x00\x00t\x05\x00\x00\x00supert\x0b\x00\x00\x00MyClassInitt\x08\x00\x00\x00__init__(\x01\x00\x00\x00t\x04\x00\x00\x00self(\x00\x00\x00\x00(\x00\x00\x00\x00s\x07\x00\x00\x00<stdin>R\x02\x00\x00\x00\x02\x00\x00\x00s\x02\x00\x00\x00\x00\x01q\x11\x85q\x12Rq\x13c__builtin__\n__main__\nh\x0eNN}q\x14tq\x15Rq\x16utq\x17Rq\x18.'

BTW: dill punts to pickle for certain cases involving classes (primarily), and pickle throws an AssertionError as one of the 3-4 errors that it might evoke. Why there's not just a PicklingError I don't know… that might be more preferable.

Mike McKerns
  • 33,715
  • 8
  • 119
  • 139