2

I'm trying to unpickle objects pickled in python3. This works in python3 but not in python2. The issue can be reproduced down to pickle protocol 0. Example code:

import pickle
import collections

o = collections.OrderedDict([(1,1),(2,2),(3,3),(4,4)])
f = open("test.pkl", "wb")
pickle.dump(o, f, 0)
f.close()

This results in the following pkl file:

python2:

ccollections
OrderedDict
p0
((lp1
(lp2
I1
aI1
aa(lp3
I2
aI2
aa(lp4
I3
aI3
aa(lp5
I4
aI4
aatp6
Rp7

python3:

cUserString
OrderedDict
p0
(tRp1
L1L
L1L
sL2L
L2L
sL3L
L3L
sL4L
L4L
s.

When I try to load the pickle file created in python3 from python2 I'm getting the following exception:

Python 2.7.6 (default, Jun 22 2015, 17:58:13) 
[GCC 4.8.2] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import pickle
>>> f = open("test.pkl", "rb")
>>> p = pickle.load(f)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/lib/python2.7/pickle.py", line 1378, in load
    return Unpickler(file).load()
  File "/usr/lib/python2.7/pickle.py", line 858, in load
    dispatch[key](self)
  File "/usr/lib/python2.7/pickle.py", line 1090, in load_global
    klass = self.find_class(module, name)
  File "/usr/lib/python2.7/pickle.py", line 1126, in find_class
    klass = getattr(mod, name)
AttributeError: 'module' object has no attribute 'OrderedDict

However changing first line in the pickle file from UserString to collections sort of solves the problem.

Python 2.7.6 (default, Jun 22 2015, 17:58:13) 
[GCC 4.8.2] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import pickle
>>> f = open("test.pkl", "rb")
>>> p = pickle.load(f)
>>> p
OrderedDict([(1L, 1L), (2L, 2L), (3L, 3L), (4L, 4L)])

Is this a bug in pickle in python3?

Milosz Wasilewski
  • 39
  • 2
  • AFAIK, you should only pickle and unpickle using the exact same major.minor version. Python3 has data types not present in Python2. If you need to cross pickle, you should serialise to common format, such as XML or JSON – Alastair McCormack Nov 06 '15 at 14:23
  • That is probably the right thing to do. Unfortunately I'm not controlling the code that generates the pickle file. I only need to read from it :( – Milosz Wasilewski Nov 09 '15 at 10:18

1 Answers1

1

Make sure you import collections in Python 2. This code works for me:

Python 3 - do the pickling:

import pickle
import collections

o = collections.OrderedDict([(1,1),(2,2),(3,3),(4,4)])
with open('/home/bo/Desktop/test.pkl', 'wb') as f:
    pickle.dump(o, f, 2)

Python 2 - do the unpickling:

import pickle
import collections

with open('/home/bo/Desktop/test.pkl', 'rb') as f:
    o = pickle.load(f)

When I do that I can read o with no issue:

>>> o
0: OrderedDict([(1, 1), (2, 2), (3, 3), (4, 4)])
bbayles
  • 4,389
  • 1
  • 26
  • 34
  • Doesn't work for me. I'm wondering what am I doing differently. Here are my python versions: Python 2.7.6 Python 3.4.3 – Milosz Wasilewski Nov 06 '15 at 17:33
  • Did you try my exact code snippets? The error you're getting indicates it's a namespace problem, not a pickle version problem. If you did `from collections import OrderedDict` in the source of the pickle file, do the same for the destination. – bbayles Nov 07 '15 at 00:04
  • Yes, I used this exact code. Here it is: https://gist.github.com/mwasilew/22f78bafc707127b32a9 and https://gist.github.com/mwasilew/2220b3c767af4da0e93f. This still gives the same error as before. – Milosz Wasilewski Nov 09 '15 at 09:56
  • Very strange. Which OS is this? Can you try a separate Py3 install? – bbayles Nov 09 '15 at 15:48
  • Checked on Ubuntu 14.04 and some CentOS. Also checked on Debian8 - same result everywhere. Slightly different minor versions of python don't seem to affect the problem. – Milosz Wasilewski Nov 10 '15 at 22:39
  • I think [this bug](https://bugs.python.org/issue18473) is the real issue... so I think you're right about it being a Python bug. – bbayles Nov 15 '15 at 16:34