keep a subprocess alive and keep giving it commands python 3

Question

The code from the accepted answer to Keep a subprocess alive and keep giving it commands? Python isn't working in python 3.6 (it just hangs). Is there a way to fix it?

from subprocess import Popen, PIPE

# Run "cat", which is a simple Linux program that prints it's input.
process = Popen(['/bin/cat'], stdin=PIPE, stdout=PIPE, encoding='utf-8', universal_newlines=True)
process.stdin.write('Hello\n')
print(repr(process.stdout.readline())) # Should print 'Hello\n'
process.stdin.write(b'World\n')
print(repr(process.stdout.readline())) # Should print 'World\n'

# "cat" will exit when you close stdin.  (Not all programs do this!)
process.stdin.close()
print('Waiting for cat to exit')
process.wait()
print('cat finished with return code %d' % process.returncode)

Where is it hanging? What OS are you running on? Can you confirm that `cat` is actually executing? — Daniel Pryden, Nov 27 '17 at 00:13
Also: if this code works on Python 2.x and fails on 3.x, I'd assume it's a bytes-vs-unicode thing. Try specifying `encoding='utf-8'` in the `Popen` constructor. — Daniel Pryden, Nov 27 '17 at 00:15
Oh, and if you're on a non-Unix OS, you'll definitely need `universal_newlines=True` in the `Popen` constructor as well. — Daniel Pryden, Nov 27 '17 at 00:17
mac os 10.12 Anaconda python 2.7 hangs on readline. unfortunately adding encoding doesn't help — maxymoo, Nov 27 '17 at 00:17
I would next try `strace` to watch what's actually going on between the processes. It might be a stdio buffer: does explicitly flushing the process's stdin help? — Daniel Pryden, Nov 27 '17 at 00:21

score 4 · Answer 1 · answered Nov 27 '17 at 00:34

As per @DanielPryden's suggestion, you need to explicitly flush the stdin:

from subprocess import Popen, PIPE

# Run "cat", which is a simple Linux program that prints it's input.
process = Popen(['/bin/cat'], stdin=PIPE, stdout=PIPE)
process.stdin.write(b'Hello\n')
process.stdin.flush()
print(repr(process.stdout.readline())) # Should print 'Hello\n'
process.stdin.write(b'World\n')
process.stdin.flush()
print(repr(process.stdout.readline())) # Should print 'World\n'

# "cat" will exit when you close stdin.  (Not all programs do this!)
process.stdin.close()
print('Waiting for cat to exit')
process.wait()
print('cat finished with return code %d' % process.returncode)

score 1 · Accepted Answer · answered Nov 27 '17 at 00:37

Between cat and your program is a buffer, most likely in the libc stdio implementation on the Python side. You need to flush this buffer to ensure cat has seen the bytes you wrote before putting your process to sleep waiting for cat to write some bytes back.

You can do this explicitly with a process.stdin.flush() call, or you can do it implicitly by disabling the buffer. I think the explicit form is probably better here: it's simple and clearly correct.

AAM111 · Answer 3 · 2017-11-27T00:28:29.580

0

In Python 3.6 (and in other versions as far as I know), process.stdin.write takes a byte object, not a string.

process.stdin.write(b'Hello\n')

EDIT: This answer was invalidated after the edit to the question. (It added an encoding to the Popen constructor.)

edited Nov 27 '17 at 00:28

answered Nov 27 '17 at 00:14

AAM111

1,178
3
19
39

Actually, this is printing `5` for me, so I'm probably wrong. – AAM111 Nov 27 '17 at 00:14
Really? I get `TypeError: a bytes-like object is required, not 'str'` on `process.stdin.write('test\n')`. This is in IDLE 3.6.2 – AAM111 Nov 27 '17 at 00:20
are you sure your're running python 3? – maxymoo Nov 27 '17 at 00:23
It takes a Unicode string if you set an encoding, or a bytestring otherwise. – Daniel Pryden Nov 27 '17 at 00:23
@maxymoo Yes. I started IDLE with `idle3`. – AAM111 Nov 27 '17 at 00:24
@DanielPryden How do you set an encoding? – AAM111 Nov 27 '17 at 00:24
1

@OldBunny2800: with the `encoding` parameter of the `Popen` constructor. Use `help(subprocess.Popen)` for more information. – Daniel Pryden Nov 27 '17 at 00:25

keep a subprocess alive and keep giving it commands python 3

3 Answers3