2

I have a python process(say abc.py) which prints a latin1 encoded pickle output which contains \n

This is the code which generates the binary pickle output

byte_array = pickle.dumps(result)    

#Windows uses utf-8 encoding by default
# http://stackoverflow.com/a/4374457/2073595
# http://stackoverflow.com/a/26220050/1873328
sys.stdout = io.TextIOWrapper(sys.stdout.detach(), encoding='latin-1')
print(byte_array.decode('latin-1'), end='', flush=True)

The output is in the following format

...00USR_RDER_TYPEq\x06X\n\x00\x00\x00VERSION_.....

Note that this binary string contains \n

I have another sublime plugin which uses suprocess.Popen to run the above script and fetch the pickled output using communicate

db_process =  subprocess.Popen('python -u abc.py',stdout=subprocess.PIPE,cwd=scripts_folder,shell=True,universal_newlines=False)
try:      
  output, errs = db_process.communicate(timeout=15)
except TimeoutExpired:
  db_process.kill()
  output, errs = db_process.communicate()

#unpickle the output
obj = pickle.loads(output)

My problem is that in the variable output, \n is changed to \r\n, i.e the above pickle output will be received as 

...00USR_RDER_TYPEq\x06X\r\n\x00\x00\x00VERSION_....

This leads to problems in un-pickling. I cannot replace the occurences as it is a binary string. How do I solve this problem?

PS: I am using python 3.4

Tapan Nallan
  • 1,762
  • 3
  • 17
  • 37

1 Answers1

2

I do NOT know how to remove the conversion from \n to \r\n.

But you can encode and decode it so that there are no spaces or they do not matter.

import base64
byte_array = pickle.dumps(result)
print(base64.b64encode(byte_array))

and then you need to revert the encoding.

import base64
obj = pickle.loads(base64.b64decode(output))
User
  • 14,131
  • 2
  • 40
  • 59