1

I am trying to convert a Python 2 code into Python 3. I used the pickle.loads function in Python 2.7, which according to its documentation does the following (https://docs.python.org/2.7/library/pickle.html):

pickle.loads(string)
Read a pickled object hierarchy from a string. Characters in the 
string past the pickled object’s representation are ignored.

However its behavior changes in Python 3 (https://docs.python.org/3/library/pickle.html):

pickle.loads(bytes_object, *, fix_imports=True, encoding="ASCII", errors="strict")
Read a pickled object hierarchy from a bytes object and return the 
reconstituted object hierarchy specified therein.

In the database, I have a string x which was the output of pickle.dumps(obj) performed in Python 2. I want to retrieve obj in Python 3. When I do pickle.loads(x), I receive the following error:

a bytes-like object is required, not 'str'

Changing pickle.loads(x) into pickle.loads(bytes(x, 'utf-8')) instead gives the following error:

invalid load key, '\x5c'.

How can I obtain obj from x in Python 3?

Irvan
  • 439
  • 4
  • 19
  • If you're trying to read a pickle created by Python 2 into Python 3, you need to set the right protocol: https://docs.python.org/3/library/pickle.html#data-stream-format – jonrsharpe Apr 22 '19 at 09:30
  • [This article](https://rebeccabilbro.github.io/convert-py2-pickles-to-py3/) outlines the entire process by which one would do this, including gotchas. – Alec Apr 22 '19 at 09:32
  • @jonrsharpe according to its documentation: The protocol version of the pickle is detected automatically, so no protocol argument is needed. Is it still necessary to explicitly set the protocol when I call python.loads? – Irvan Apr 22 '19 at 09:33
  • @alec_a The article talks mostly about `pickle.load` instead of `pickle.loads`. I couldn't resolve my problem by trying a similar fix for `pickle.loads`. – Irvan Apr 22 '19 at 09:34
  • http://blog.yannisassael.com/2016/03/open-python-2-pickle-python-3/ – Alec Apr 22 '19 at 09:35
  • @alec_a the string `x` in my scenario is retrieved from the database instead of from a file. I don't know how to use `pickle.load` in place of `pickle.loads`. Setting the encoding to latin1 does not resolve my issue, and I still receive the `invalid load key, '\x5c'` error. – Irvan Apr 22 '19 at 09:38

1 Answers1

2

Change pickle.loads(x) to pickle.loads(bytes(x, 'latin-1')).

Change pickle.dumps(o) to str(pickle.dumps(o, protocol=0), 'latin-1').

Irvan
  • 439
  • 4
  • 19
  • Or leveraging the internal of python, passing the `encode` arg would do it. `pickle.load(x, encoding='latin1')` Checkout this answer https://stackoverflow.com/questions/28218466/unpickling-a-python-2-object-with-python-3 – Hrimiuc Paul Sep 22 '22 at 18:26