Dumping a subclass of gtk.ListStore using pickle

Question

I am trying to dump a custom class using pickle. The class was subclassed from gtk.ListStore, since that made it easier to store particular data and then display it using gtk. This can be reproduced as shown here.

import gtk
import pickle
import os

class foo(gtk.ListStore):
    pass

if __name__=='__main__':
    x = foo(str)
    with open(os.path.expandvars('%userprofile%\\temp.txt'),'w') as f:
        pickle.dump(x,f)

The solution that I have tried was to add a __getstate__ function into my class. As far as I understand the documentation, this should take precedence for pickle so that it no longer tries to serialize the ListStore, which it is unable to do. However, I still get an identical error from pickle.dump when I try to pickle my object. The error can be reproduced as follows.

import gtk
import pickle
import os

class foo(gtk.ListStore):
    def __getstate__(self):
        return 'bar'

if __name__=='__main__':
    x = foo(str)
    with open(os.path.expandvars('%userprofile%\\temp.txt'),'w') as f:
        pickle.dump(x,f)

In each case, pickle.dump raises a TypeError, "can't pickle ListStore objects". Using print statements, I have verified that the __getstate__ function is run when using pickle.dump. I don't see any hints as to what to do next from the documentation, and so I'm in a bit of a bind. Any suggestions?

score 1 · Answer 1 · edited Aug 20 '11 at 10:48

With this method you can even use json instead of pickle for your purpose.

Here is a quick working example to show you the steps you need to employ to pickle "unpicklable types" like gtk.ListStore. Essentially you need to do a few things:

Define __reduce__ which returns a function and arguments needed to reconstruct the instance.
Determine the column types for your ListStore. The method self.get_column_type(0) returns a Gtype, so you will need to map this back to the corresponding Python type. I've left that as an exercise - in my example I've employed a hack to get the column types from the first row of values.
Your _new_foo function will need to rebuild the instance.

Example:

import gtk, os, pickle

def _new_foo(cls, coltypes, rows):
    inst = cls.__new__(cls)
    inst.__init__(*coltypes)
    for row in rows:
        inst.append(row)
    return inst

class foo(gtk.ListStore):

    def __reduce__(self):
        rows = [list(row) for row in self]
        # hack - to be correct you'll really need to use 
        # `self.get_column_type` and map it back to Python's 
        # corresponding type.
        coltypes = [type(c) for c in rows[0]]
        return _new_foo, (self.__class__, coltypes, rows)

x = foo(str, int)
x.append(['foo', 1])
x.append(['bar', 2])

s = pickle.dumps(x)

y = pickle.loads(s)
print list(y[0])
print list(y[1])

Output:

['foo', 1]
['bar', 2]

Rosh Oxymoron · Answer 2 · 2011-05-11T19:35:34.800

When you subclass object, object.__reduce__ takes care of calling __getstate__. It would seem that since this is a subclass of gtk.ListStore, the default implementation of __reduce__ tries to pickle the data for reconstructing a gtk.ListStore object first, then calls your __getstate__, but since the gtk.ListStore can't be pickled, it refuses to pickle your class. The problem should go away if you try to implement __reduce__ and __reduce_ex__ instead of __getstate__.

>>> class Foo(gtk.ListStore):
...     def __init__(self, *args):
...             super(Foo, self).__init__(*args)
...             self._args = args
...     def __reduce_ex__(self, proto=None):
...             return type(self), self._args, self.__getstate__()
...     def __getstate__(self):
...             return 'foo'
...     def __setstate__(self, state):
...             print state
... 
>>> x = Foo(str)
>>> pickle.loads(pickle.dumps(x))
foo
<Foo object at 0x18be1e0 (__main__+Foo-v3 at 0x194bd90)>

As an addition, you may try to consider other serializers, such as json. There you take full control of the serialiazaton process by defining how custom classes are to be serialized yourself. Plus by default they come without the security issues of pickle.

Dumping a subclass of gtk.ListStore using pickle

2 Answers2