Passing a python object from a python2 script to python3 script

Question

I ran into an issue where my current traffic generator: Warp17 has generated python code that is only python2 compatible. We are moving all else to python3 so i have created a cli tool that wraps our warp17 api. Is there a better way to pass information from a subscript than just printing the output and loading it in json?

#!/usr/bin/python

import python2_api

if __name__ == "__main__":
  do_some_argparse()
  result = dict(run_some_code_on_args())
  print(result)

def run_warp_cli(*args):
  result = subprocess.Popen('warp_cli', args)
  return json.loads(result)

This is basically what I am doing. I was curious if there was any interesting way to avoid doing this and get back an actual python dictionary object etc.

Probably not. You can only transfer the data in the dictionary, not the structure itself, since the implementations of dict are different in ABI between python 2 and 3. Serialization, whether to JASON or some other mutually agreeable format seems like a good way to do that. — Mad Physicist, Aug 16 '18 at 14:51
That would be essentially the same thing, just a slightly different storage format is all. It's probably possible since I doubt that `dict` would ever pickle the actual hash table: hash values change between runs even of the same interpreter. — Mad Physicist, Aug 16 '18 at 14:57
Yes you can, although you have to be careful with data types that changed between versions: https://stackoverflow.com/q/28218466/2988730 — Mad Physicist, Aug 16 '18 at 15:03

score 1 · Accepted Answer · answered Aug 16 '18 at 15:12

The binary representation of dictionaries, and in fact objects in general, has changed quite a bit between Python 2 and Python 3. That means that you will not be able to use the objects directly between versions of Python. Some form of serialization to a mutually acceptable format will be necessary.

The JSON format you are using fits this criterion. I would recommend sticking with it as it is flexible, human readable and quite general.

If space becomes an issue, you can look into pickling as an alternative, but I would not recommend it as a first choice. While pickle.load can understand Python 2 data just fine, there are many caveats to keep in mind with different interpretations of datatypes between Python versions, especially bytes: Unpickling a python 2 object with python 3

Here is a resource that may help you with some common errors from unpicking on the Python 3 side: TypeError: a bytes-like object is required, not 'str' when opening Python 2 Pickle file in Python 3

Passing a python object from a python2 script to python3 script

1 Answers1