Python: Can't pickle type X, attribute lookup failed

Question

I am trying to pickle a namedtuple:

from collections import namedtuple
import cPickle

class Foo:

    Bar = namedtuple('Bar', ['x', 'y'])

    def baz(self):
        s = set()
        s.add(Foo.Bar(x=2, y=3))
        print cPickle.dumps(s)

if __name__ == '__main__':
    f = Foo()
    f.baz()

This produces the following output:

Traceback (most recent call last):
  File "scratch.py", line 15, in <module>
    f.baz()
  File "scratch.py", line 11, in baz
    print cPickle.dumps(s)
cPickle.PicklingError: Can't pickle <class '__main__.Bar'>: attribute lookup __main__.Bar failed

What am I doing wrong? Is the problem that Bar is a member of Foo? (Moving the definition of Bar to the top level solves the problem, although I'm still curious why this happens.)

Is @DaveButler 's answer correct? Can anyone confirm or deny? — Nathan majicvr.com, Sep 23 '18 at 16:42

score 41 · Accepted Answer · edited Apr 19 '18 at 04:00

41

Yes, the fact that it's a class member is a problem:

>>> class Foo():
...     Bar = namedtuple('Bar', ['x','y'])
...     def baz(self):
...         b = Foo.Bar(x=2, y=3)
...         print(type(b))
...
>>> a = Foo()
>>> a.baz()
<class '__main__.Bar'>

The problem is that when namedtuple() returns a type object, it isn't aware of the fact that it's being assigned to a class member - and thus, it tells the type object that its type name should be __main__.Bar, even though it should really be __main__.Foo.Bar.

edited Apr 19 '18 at 04:00

Dan Nissenbaum

13,558
21
105
181

answered Jan 13 '11 at 05:23

Amber

507,862
82
626
550

2

`namedtuple` just doesn't play nice with classes. You might be able to write a custom `__getstate__` for the object which would take care of it manually. – Amber Jan 13 '11 at 05:25
Would an early bind grab that? – motoku Jun 05 '11 at 02:58
1

How could I use __getstate__? I don't know how to implement that. At the moment I'm working with a temporary solution that transfers a list of named tuples to a dictionary. – So S Apr 18 '17 at 13:41

score 17 · Answer 2 · answered Jan 13 '11 at 10:22

Nesting classes makes pickle fail, since it relies on the path of the object inside your application to reconstruct it later.

The immediate solution is to not nest classes, i.e. move Bar definition to outside Foo. Code will work all the same.

But a better thing to do is to not use pickle at all to store data. Use some other serialization format, like json, or a database, like sqlite3.

You have just hit one of the many inconveniences of pickle, if you change your code, move things around, or sometimes make small structural changes, your data becomes unloadable.

Besides that, pickle has other disadvantages: It is slow, unsecure, python-only...

score 9 · Answer 3 · answered Dec 20 '14 at 19:42

9

Using dill in place of pickle here will allow this to work

answered Dec 20 '14 at 19:42

Maximilian

7,512
3
50
63

4

`import dill; dill.dumps(x)` – Maximilian Sep 24 '18 at 18:07
1

Also this doesn't work for me. I get: Can't pickle : it's not found as __main__.RoadAttributesPartition – Zioalex May 16 '19 at 12:16

score 2 · Answer 4 · answered Oct 27 '19 at 20:39

2

The solution here is to move your named tuple definition to the module level, then pickle works. A detailed answer is provided here:

How to pickle a namedtuple instance correctly

answered Oct 27 '19 at 20:39

David Parks

30,789
47
185
328

Python: Can't pickle type X, attribute lookup failed

4 Answers4

Linked