In Pandas, how read a Pickle file that throws Unpickling Error: invalid load key?

Question

I'm reading a pickle file with:

pandas.read_pickle('data/file.pickle')

and it throws this error:

UnpicklingError: invalid load key, '\x00'.

Note that I've seen other threads on how to solve this problem when saving the pickle file, but in my case I just need to open this specific dataframe.

Also the pickle file could contain some special characters.

Possible duplicate of [What causes the error "\_pickle.UnpicklingError: invalid load key, ' '."?](https://stackoverflow.com/questions/33049688/what-causes-the-error-pickle-unpicklingerror-invalid-load-key) — dfundako, Mar 27 '18 at 20:13
@roganjosh it happens the same thing. It seems to be some kind of encoding problem. — Henrique Nader, Mar 27 '18 at 20:15
@roganjosh it was created with pandas as well: dataframe.to_pickle(...) — Henrique Nader, Mar 27 '18 at 20:19
Can you provide a reproducer others can use to create such a file? — Charles Duffy, Mar 27 '18 at 20:20
@CharlesDuffy I've used the same code to open many pickle files, it it works great, so I think the problem is related to the content in file, maybe the encoding (since I don't know what '\x00' means). Is there a way for me to upload the file here? I could. — Henrique Nader, Mar 27 '18 at 20:26
I could open the file with Sublime. It's only zeros. Does it mean it's corrupted or something like this? — Henrique Nader, Mar 27 '18 at 21:42

score 7 · Answer 1 · answered Jul 26 '19 at 17:07

7

One of the possible explanation is pickling with compression. On my system, reading files compressed with either xz or gzip throws exception of invalid load key if I do not specify compression. In similar case, both zip and bz2 raise different exceptions.

I recommend to try one of those:

pandas.read_pickle('data/file.pickle', 'xz')
pandas.read_pickle('data/file.pickle', 'gzip')

answered Jul 26 '19 at 17:07

heavelock

364
4
9

Extremely helpful answer :) Solved our problem right away. Saved us some lines of code too, since our target is a Pandas DataFrame anyways. Thanks! – Aaron Bell Jul 27 '22 at 14:18

In Pandas, how read a Pickle file that throws Unpickling Error: invalid load key?

1 Answers1

Linked