Python Pickle crashing on load in Windows

Question

I've got a program that uses Pickle to save and load objects from disk. The pickle saving and loading looks like this:

def saveData(self,obj):
    f = open(os.path.join(self.directory,obj.name),'wb')
    pickle.dump(obj, f)

def loadData(self,fname):
    f = open(os.path.join(self.directory,fname),'rb')
    ret = pickle.load(f)
    return ret

And I have a test method that simply makes an object, saves it, then immediately loads it. In Linux, the object works fine, but in Windows, the program crashes when it tries to load the object. I've re-made the pickled file in Windows, so it isn't trying to open the Linux object dump (although if there's a way to do cross-platform pickling, I would like to know about it)

What could be causing this program to crash in Windows but not Linux?

How does it crash? What exception do you get? On which line? — Ella Sharakanski, Apr 24 '15 at 17:59
Also, you should always close the file when you're done using it. Using the `with` statement do to that is recommended. — Ella Sharakanski, Apr 24 '15 at 18:01
No exception, I get a "Python.exe has stopped working" message in windows. — Matthew Fournier, Apr 24 '15 at 18:02
Using a pdb please check on which line it happens - the `load` or the `open`. Also, please provide a simple version of the object you're pickling (the simplest version that doesn't work). — Ella Sharakanski, Apr 24 '15 at 18:07

score 1 · Answer 1 · answered Apr 24 '15 at 17:52

Pickle basically only stores the attributes of an object and the classname to recreate it.

If you happen to have a C/C++ wrapped object, the attributes might contain pointers to memory locations (aka C-Pointers). So if you pickle that object, destory the old object and restore it, the pointers point somewhere invalid. If those are used during unpickling, you get a crash. And the crash will happen on one OS or the other depending on how the memory manager reused the address. Windows seems to be more eager to free things.

So, does this happen for some extension object? It should not happen for a pure python class.

score 1 · Answer 2 · answered May 05 '21 at 12:44

1

I had a similar issue that my Python interpreter simply crashed when trying to load a pickled data frame with Pandas pd.read_pickle(). I solved it by updating Pandas with pip install -U pandas (now version 1.2.4).

answered May 05 '21 at 12:44

stefanbschneider

5,460
8
50
88

score 0 · Answer 3 · answered Jul 10 '20 at 15:02

The issue here is related to the differences in the way Python handles certain objects (e.g. paths) depending on the OS.

The nature of the underlying issue is described in this answer.

In short: pickled files are not transferable between OSes (most of the time).

Python Pickle crashing on load in Windows

3 Answers3