cPickle and open('w') on Windows

Question

So I just spent a long time processing and writing out files for a project I'm working on. The files contain objects pickled with cPickle. Now I'm trying to load the pickled files and I'm running into the problem: "Can't import module ...". The thing is, I can import the module directly from the python prompt just fine.

I started noticing that my code had the problem reading the file (getting EOF error) and I noted that I was reading it with open('file','r'). Others noted that I need to specify that it's a binary file. I don't get the EOF error anymore, but now I'm getting this error.

It seems to me that I've screwed up the writing of my files initially by writing out with 'w' and not 'wb'.

The question I have is, is there a way to process the binary file and fix what 'w' changed? Possibly by searching for line returns and changing them (which is what I think the big difference is between 'w' and 'wb' on Windows).

Any help doing this would be amazing, as otherwise I will have lost weeks of work. Thanks.

Have you tried `with open('foo', 'wb') as f, open('bar', 'r') as g: f.write(g.read)`? — Joel Cornett, Apr 02 '15 at 04:05
open('bar', 'r') only reads the first part of the file (that's what was causing the EOF error). open('bar','rb') reads the whole file. — Nate Diamond, Apr 02 '15 at 04:08
What does a hexdump of the file reveal? Do you see a lot of \r\n's? — Joel Cornett, Apr 02 '15 at 04:14
Yes. I did `open('bar', 'rb')`, then did a bunch of `readline()` and i see lots of `\r\n`. Should I just parse the file, replacing \r\n with something else? — Nate Diamond, Apr 02 '15 at 04:31
Yeah I would try that. It looks like that's the primary difference between text and binary mode anyway. See this answer: http://stackoverflow.com/a/9644141/1142167 — Joel Cornett, Apr 02 '15 at 14:22

score 0 · Answer 1 · answered Apr 02 '15 at 14:25

I found the answer here. It talks about a solution to the same problem having, but not before outlining the traditional solution in python 2 (to all those that do this, thank you).

The solution comes down to this:

data = open(filename, "rb").read()
newdata = data.replace("\r\n", "\n")
if newdata != data:
    f = open(filename, "wb")
    f.write(newdata)
    f.close()

Basically, just replace all instances of "\r\n" with "\n". It seems to have worked well, I can now open the file and unpickle it just fine.

cPickle and open('w') on Windows

1 Answers1