ImportError: No module named copy_reg pickle

Question

I'm trying to unpickle an object stored as a blob in a MySQL database. I've manually generated and stored the pickled object in the database, but when I try to unpickle the object, I get the following rather cryptic exception:

ImportError: No module named copy_reg

Any ideas as to why this happens?

Method of Reproduction

Note: Must do step 1 on a Windows PC and steps 3 and 4 on a Linux PC.

1) On a Windows PC:

file = open("test.txt", "w")
thing = {'a': 1, 'b':2}
cPickle.dump(thing, file)

2) Manually insert contents of text.txt into blob field of MySQL database running on linux

3) In Python running on a linux machine, fetch the contents of column from MySQL

4) Assuming that you put the contents of the blob column into a variable called data, try this:

cPickle.loads(rawString)

@Stephen Edmonds: Since there's no code, I was forced to assume what code you were using. Sadly, the code I assumed worked perfectly, so I couldn't reproduce your problem. If I had your code, instead of mine, I might have been able to help. — S.Lott, Feb 17 '09 at 11:47
@ S.Lott: I've now added code snippets, but as you can see, they don't really help much. The main reason why the issue is hard to reproduce is that you need to pickle the object on Windows, save it to a file and then try and unpickle it on Linux. — Stephen Edmonds, Feb 17 '09 at 12:31
@Stephen Edmonds: I note that none of the Windows-Linux stuff in the original question. It turns out that providing the code also included useful, interesting facts. — S.Lott, Feb 17 '09 at 13:32
The extra weirdness with this is, that even if 'copy_reg' exists inside the sys.modules you get the cryptic ImportError: no module named copy_reg. — schlenk, Aug 18 '10 at 11:13
This issue is even more frustrating to deal with when git changes the line endings on windows machines — Maciej, Mar 27 '19 at 14:53

Stephen Edmonds · Accepted Answer · 2009-02-17T12:33:21.797

24

It seems this might be caused by my method of exporting the pickled object.

This bug report seens to suggest that my issue can be resolved by exporting to a file writen in binary mode. I'm going to give this a go now and see if this solves my issue.

UPDATE: This works. The solution is to make sure you export your pickled object to a file open in binary mode, even if you are using the default protocol 0 (commonly referred to as being "text")

Correct code based on orignal example in question:

file = open("test.txt", 'wb')
thing = {'a': 1, 'b':2}
cPickle.dump(thing, file)

edited Feb 17 '09 at 12:33

answered Feb 17 '09 at 10:52

Stephen Edmonds

948
1
9
25

Again, what exactly would code show? How would it be of benifit to this answer? – Stephen Edmonds Feb 17 '09 at 12:32
If just want to mention that `No module named copy_reg pickle` can be also any other module name like in my case. This might be confusing during debugging. – holzkohlengrill Nov 16 '16 at 10:10

score 16 · Answer 2 · answered Apr 17 '09 at 13:50

16

Also, simply running dos2unix (under linux) over the (windows-created) pickle file solved the problem for me. (Haven't tried the open mode 'wb' thing.) Dan

answered Apr 17 '09 at 13:50

This shoiuld be an asnwer – 00__00__00 May 30 '18 at 08:32

score 3 · Answer 3 · answered Aug 07 '09 at 14:42

just an interactive python session to show that you don't need any particular code to have this problem:

do something like this on a windows machine

Python 2.4.1 (#65, Mar 30 2005, 09:13:57) [MSC v.1310 32 bit (Intel)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> import pickle, re
>>> empty_string = re.compile("^$")
>>> pickle.dump([empty_string,1,1.23,'abc'], file('m:/mario/test-b.dump','wb'))
>>> pickle.dump([empty_string,1,1.23,'abc'], file('m:/mario/test-t.dump','wt'))
>>>

and then try to retrieve the data from a linux box

Python 2.6.2 (release26-maint, Apr 19 2009, 01:56:41) 
[GCC 4.3.3] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import pickle
>>> pickle.load(file('/home/mario/.gvfs/transfer on 192.168.0.4/mario/test-b.dump'))
/usr/lib/python2.6/pickle.py:1124: DeprecationWarning: The sre module is deprecated, please import re.
  __import__(module)
[<_sre.SRE_Pattern object at 0xb7d42420>, 1, 1.23, 'abc']
>>> pickle.load(file('/home/mario/.gvfs/transfer on 192.168.0.4/mario/test-t.dump'))
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/lib/python2.6/pickle.py", line 1370, in load
    return Unpickler(file).load()
  File "/usr/lib/python2.6/pickle.py", line 858, in load
    dispatch[key](self)
  File "/usr/lib/python2.6/pickle.py", line 1090, in load_global
    klass = self.find_class(module, name)
  File "/usr/lib/python2.6/pickle.py", line 1124, in find_class
    __import__(module)
ImportError: No module named sre
>>>

the error message can be even more confusing if you are just pickling base types. this is what I get with the list [12, 1.2, '']:

ValueError: insecure string pickle

score 2 · Answer 4 · answered Oct 30 '15 at 10:47

As mentioned in the other answer use

dos2unix originalPickle.file outputPickle.file

OR use the tr command like below (removes carriage returns and ctrl-z)

  tr -d '\15\32' < originalPickle.file > outputPickle.file

OR use awk (gawk or nawk if its old versions)

  awk '{ sub("\r$", ""); print }' originalPickle.file > outputPickle.file

OR if you can recreate a pickle file in linux, use that.

score 1 · Answer 5 · answered Oct 19 '14 at 02:09

1

Another thing going on here is that you don't seem to have closed the file after dumping the pickle to it. the error reported here can sometimes be caused (whether on a windows machine or otherwise) by not closing the file.

answered Oct 19 '14 at 02:09

dan

21
1

score 1 · Answer 6 · answered Aug 28 '18 at 02:40

1

my issue：

with open('model.pkl', 'rb') as f:
    subsection_struct_model = pickle.load(f)

when i get the model.pkl from windows run this code in my mac,this issue is comming

solve:

dos2unix model.pkl

is ok!

answered Aug 28 '18 at 02:40

merlin.chan

11
1

Statham · Answer 7 · 2018-12-27T09:25:06.627

This happen because in Windows, new line symbol is stored as "\r\n", and that in Linux is "\n",so what you should do is to read the pickle file line by line, replace "\r\n" with "\n", and write it back to the file.

Unlike the dos2unix approach will skip binary file, this also works well for binary pickle file.

Code is simple:

#run this code on Linux platform

DIR="your/dir/to/file"

pickle.load(open(DIR,"rb"))
# ImportError: No module named copy_reg pickle

a=open(DIR,"rb").readlines() #read pickle file line by line

a=map(lambda x:x.replace("\r\n","\n"),a) # replace \r\n with \n

with open(DIR,"wb") as j: #write back to file in binary mode
    for i in a:
        j.write(i)         
pickle.load(open(DIR,"rb")) 
#Now it works well

Also, you can use normal write mode, just change "wb" to "w" and "rb" to "e".

Straightforward answer. Works! Thanks! – Xonshiz Jul 22 '19 at 08:39 — Xonshiz, Jul 22 '19 at 08:39

score 1 · Answer 8 · answered May 25 '20 at 09:24

I ran into this problem, because of git changing the line endings on windows.

I solved the problem by adding a .gitattributes file to my repo:

# Pickle files (for testing) should always have UNIX line endings. 
*.pkl text eol=lf

See also https://github.com/actions/checkout/issues/135

score 0 · Answer 9 · answered Feb 16 '16 at 17:55

0

The pickle load may not be looking in the same location as your python script. Sometimes the directory changes on the basis of your application. Just before you load the pickle, print a os.getcwd() to work out a solution.

answered Feb 16 '16 at 17:55

Arslán

1,711
2
12
15

ImportError: No module named copy_reg pickle

9 Answers9

Linked