How can I send a byte-object over a pipe in python?

Question

I am trying to send a numpy array over stdout from one process to another.

Sending it over the pipe requires me to convert it to a string. On the other side, I receive a byte object. This byte object now has the original byte string encapsulated.

I find it now impossible to restore the original byte object as a byte object. If I decode the byte object, I receive a string which is incompatible with all the restoring functions I tried (np.frombuffer, pickle.loads).

server.py

import subprocess
import numpy as np
p = subprocess.Popen(['python3', 'writer.py'], stdout=subprocess.PIPE)

while 1:
    tmp = p.stdout.readline()    
    # doesn't fail but wrong size
    array = np.frombuffer(tmp, dtype=np.uint8)
    tmp = bytes.decode(tmp)
    # fails because byte object is necessary
    array = np.frombuffer(tmp, dtype=np.uint8)
    array = array.reshape([1, 3, 5, 5, 1])
    print(array.shape)

writer.py

import numpy as np
import sys
while 1:
    array = np.zeros([1, 3, 5, 5, 1], dtype=np.int8)
    string = array.tobytes()
    sys.stdout.write(str(string))
    sys.stdout.flush()

Is there anyway to convert the string into a byteobject, without encoding it? How else could this work? I want to use pipes instead of shared memory as proposed in some other solutions to keep it more simple. Furthermore, I need it to be parallel but blocking, so Pipes seemed ideal to me.

Thanks

how about you first send the size of the object to come, then read that size from the buffer in the server? — Netwave, Mar 06 '17 at 08:18
@DanielSanchez I don't quite understand, there's no buffer on the server side? There's stdout from which I can read out everything that was sent. But reading the whole string is not the problem. I want to restore the original object. — Kilsen, Mar 06 '17 at 08:23
@Kilsen have you tried in the writer `sys.stdout.write(string)` without converting to `str`? — MB-F, Mar 06 '17 at 08:25
@kazemakase Yes, this isn't possible because `stdout.write` requires a string — Kilsen, Mar 06 '17 at 08:27
@Kilsen `sys.stdout.write(b'abc')` works for me. Why does it not work for you? — MB-F, Mar 06 '17 at 08:34
@kazemakase I am on python 3, this seems to work on python 2 — Kilsen, Mar 06 '17 at 08:36
@kazemakase: weird: `Python 3.5.2 (default, Nov 17 2016, 17:05:23) [GCC 5.4.0 20160609] on linux Type "help", "copyright", "credits" or "license" for more information. >>> import sys >>> sys.stdout.write(b'abc') Traceback (most recent call last): File "", line 1, in TypeError: write() argument must be str, not bytes >>> ` — Kilsen, Mar 06 '17 at 08:42
@Kilsen Doh! This seems to depend on the terminal backend used. Running in Spyder it worked, but running in the terminal it failed for me too. Check out [this answer](http://stackoverflow.com/a/908440/3005167). — MB-F, Mar 06 '17 at 08:49
@kazemakase: thanks for your help! yes, stdout.buffer was what I was looking for. When I tried using stdout.buffer.write I got an additional byte on the receiving side for some reason. ymonad's solution is working perfectly. — Kilsen, Mar 06 '17 at 09:03

score 3 · Accepted Answer · answered Mar 06 '17 at 08:46

you can use pickle to marshal the data, and sys.stdout.buffer instead of sys.stdout to write bytes to stdout.

Reference: sys.stdout

server.py:

import subprocess
import numpy as np
import pickle
p = subprocess.Popen(['python3', 'writer.py'], stdout=subprocess.PIPE)

while 1:
  array = pickle.load(p.stdout)
  array = array.reshape([1, 3, 5, 5, 1])
  print(array.shape)

writer.py:

import numpy as np
import pickle
import sys
while 1:
  array = np.zeros([1, 3, 5, 5, 1], dtype=np.int8)
  pickle.dump(array, sys.stdout.buffer)

How can I send a byte-object over a pipe in python?

1 Answers1