Python pickle doesn't work with UserLists/UserDicts

Question

UserDicts and UserLists seem to work unstable with pickle module. How do I fix this bug:

test_pickle.py

import pickle

class UserList(list):
    pass

class UserDict(dict):
    pass

u = UserList([])
for i in range(10):
    d = UserDict()
    d.u = u
    u.append(d)

pickle.dump(u, open("ttt.pcl", 'wb'))

$ python test_pickle.py

  ... <~300 traceback lines>
  File "/usr/lib/python2.7/pickle.py", line 663, in _batch_setitems
    save(v)
  File "/usr/lib/python2.7/pickle.py", line 331, in save
    self.save_reduce(obj=obj, *rv)
  File "/usr/lib/python2.7/pickle.py", line 405, in save_reduce
    self.memoize(obj)
  File "/usr/lib/python2.7/pickle.py", line 244, in memoize
    assert id(obj) not in self.memo
AssertionError

Now, if I increase the number of elements in UserList, it gets even "better":

import pickle

class UserList(list):
    pass

class UserDict(dict):
    pass

u = UserList([])
for i in range(100):
    d = UserDict()
    d.u = u
    u.append(d)

pickle.dump(u, open("ttt.pcl", 'wb'))

$python test_pickle.py

... <more lines than my terminal can handle>
  File "/usr/lib/python2.7/pickle.py", line 649, in save_dict
    self._batch_setitems(obj.iteritems())
  File "/usr/lib/python2.7/pickle.py", line 663, in _batch_setitems
    save(v)
  File "/usr/lib/python2.7/pickle.py", line 306, in save
    rv = reduce(self.proto)
  File "/usr/lib/python2.7/copy_reg.py", line 71, in _reduce_ex
    state = base(self)
RuntimeError: maximum recursion depth exceeded while calling a Python object

@jonrsharpe Thanks for pointing to that question. A much different manifestation of essentially the same issue. — Boris Burkov, Jan 27 '14 at 19:49

score 4 · Accepted Answer · answered Jan 27 '14 at 17:30

You have some circular references.

d.u = u
u.append(d)

Pickle protocol 0 has issues with this, as you are experiencing. The simplest way to fix: specify protocol=-1:

pickle.dump(u, open("ttt.pcl", 'wb'), protocol=-1)

From the docs:

There are currently 3 different protocols which can be used for pickling.

Protocol version 0 is the original ASCII protocol and is backwards compatible with earlier versions of Python.
Protocol version 1 is the old binary format which is also compatible with earlier versions of Python.
Protocol version 2 was introduced in Python 2.3. It provides much more efficient pickling of new-style classes.

Protocol 0 is the default (in python 2) and specifying -1 means "use the highest protocol available".

Python pickle doesn't work with UserLists/UserDicts

1 Answers1