0

I'm trying to retrieve pickle data I have uploaded to an openstack object storage using openstacksdk's connection.get_object(container,object), I get a response from it, however the file body is a string, I can even save it to file with the outfile option without issues. However I would like to be able to work with it directly without having to resort to save it to file first and then loading it into pickle.

Simply using pickle's load and loads doesn't work as neither takes string objects. Is there another way to retrieve the data so I can work with the pickled data directly or is there some way to parse to string/set a config parameter on get_object()?

Xavanteex
  • 97
  • 5

2 Answers2

0

If you are using Python 3 - pickle expects a bytes-like object. The load method takes a file path, and relies on the file type to handle the providing of bytes back into pickle. When you use the loads method you need to provide it a bytes-like object, not a string, so you will need to convert the string to bytes.

Best way to convert string to bytes in Python 3?

Sean
  • 671
  • 5
  • 11
  • Seems the problem is the data is saved with the 'wb' flag in writing it. And using the suggested solutions in the answer you provided does not encode it correctly. – Xavanteex Nov 19 '20 at 22:46
0

EDIT: I found the solution, for pickled objects or any other files retrieved from openstack with openstacksdk, there are a few ways of dealing with the data without resorting to disk.

First my implemented solution was to use openstack's connection method get_object_raw:

conn = connection(foo,bar, arg**) 
pickle.loads(conn.get_object_raw('containerName', 'ObjectName').content)

.get_object_raw returns a response request object with the attribute content which is the binary file content which is the pickle content one can load with pickle.

You could also create a temporary in-memory file with io.BytesIO, and using it as the outfile argument in get_object from the connection object.

Xavanteex
  • 97
  • 5