python subprocess multiple stdin.write and stdout.read

Question

Thanks for taking time to answer the question. I am playing around with Python 3.4 and I have two simple python programs. One, a program called test.py that takes a user input and prints something.

while True:
    print("enter something...")
    x = input()
    print(x)
    time.sleep(1)

To send input to this program, I have another program that uses subprocess:

from subprocess import Popen, PIPE

cat = Popen('python test.py', shell=True, stdin=PIPE, stdout=PIPE)
cat.stdin.write("hello, world!\n")
cat.stdin.flush()
print(cat.stdout.readline())

cat.stdin.write("and another line\n")
cat.stdin.flush()
print(cat.stdout.readline())

However when I run the above program, I get an error:

enter something...

hello, world!
Traceback (most recent call last):
  File "/opt/test.py", line 9, in <module>
    x = input()
EOFError: EOF when reading a line
Exception ignored in: <_io.TextIOWrapper name='<stdout>' mode='w' encoding='UTF-8'>
BrokenPipeError: [Errno 32] Broken pipe

And if I replace test.py with a standard linux command like 'cat', things work as expected.

Is there any way I send multiple stdin writes and read multiple stdout back?

I think you missed the command **python** on Popen `Popen(['python', 'test.py'], shell=True, stdin=PIPE, stdout=PIPE)` — Ramon Moraes, Oct 24 '15 at 01:32
Typo, the program actually runs if you see that the first print works. — waka-waka-waka, Oct 24 '15 at 01:57
`cat.stdin.write("hello, world!\n")` should raise an error if you use Python 3 (`universal_newlines=True` is required to enable the text mode) therefore either your actual code is different or you are not using Python 3. — jfs, Oct 24 '15 at 13:36
unrelated: you should [`import` the module and call its functions (use `multiprocessing` if necessary) instead of running it as a subprocess](http://stackoverflow.com/a/30165768/4279) — jfs, Oct 24 '15 at 14:42

jfs · Accepted Answer · 2017-07-19T15:19:18.213

In general, you should use pexpect for interactive programs (dialog-based interactions).

Your specific issue might be caused by a python version mismatch (you think your code is executed using Python 3 while actually it might be executed using Python 2). The second issue (EOFError) is expected: either catch it in the child script or provide a signal for the child to exit (I use an empty line for that in the code example below).

Here's a Python 3 code that fails loudly on Python 2:

#!/usr/bin/env python3
import sys
from subprocess import Popen, PIPE

with Popen([sys.executable, '-u', 'test.py'], stdin=PIPE, stdout=PIPE,
           universal_newlines=True, bufsize=1) as cat:
    for input_string in ["hello, world!", "and another line", ""]:
        print(input_string, file=cat.stdin, flush=True)
        print(cat.stdout.readline(), end='')

Note:

sys.exectable is the current python executable (Python 3 in this case)
universal_newlines=True enables the text mode (otherwise, cat.stdin and cat.stdout work with bytes, not strings)
-u makes the child's output line-buffered (otherwise, you won't see anything until the child flushes its internal stdout buffer)
the with-statement closes the pipes and waits for the child process to exit.

And here's the corresponding test.py:

#!/usr/bin/env python3
import time

while True:
    x = input("enter something...")
    if not x: # exit if the input is empty
        break
    print(x)
    time.sleep(1)

Output

enter something...hello, world!
enter something...and another line
enter something...

Note: there is no new line after "enter something..."

It works but it is fragile, read Q: Why not just use a pipe (popen())? and use pexpect instead.

If the input is finite and it doesn't depend on the output then you could pass it all at once:

#!/usr/bin/env python3
import sys
from subprocess import check_output

output = check_output([sys.executable, 'test.py'],
                      input="\n".join(["hello, world!", "and another line"]),
                      universal_newlines=True)
print(output, end='')

This version requires that the child handles EOF properly:

#!/usr/bin/env python3
import time

while True:
    try:
        x = input("enter something...")
    except EOFError:
        break # no more input

    print(x)
    time.sleep(1)

The output is the same (as shown above).

Thanks so much for the detailed explanation Sebastian! :). How do I read multiple lines? using cat.stdout.readline() will hang forever? — waka-waka-waka, Oct 24 '15 at 23:50
@waka-waka-waka: both code examples already read multiple lines — jfs, Oct 24 '15 at 23:51
thanks, my question was more on the lines of, I am not sure how many lines to read, when do I break? The test program may print 1 or more lines... — waka-waka-waka, Oct 24 '15 at 23:53
`check_output()` -based solution reads all output, no matter how many lines a subprocess produces. — jfs, Oct 24 '15 at 23:56
to make it more clear, assume the case where the child does not print anything to the screen, in which case the .readline() will hang forever? — waka-waka-waka, Oct 24 '15 at 23:56
`check_output()` won't hang even if the child prints nothing. — jfs, Oct 24 '15 at 23:58
thank you, I am almost there... in my case the process has launched previously and i am using psutils and other modules to manage the process. In this case, I am using the popen object to do stdin, stdout etc. cant use check_output since the process is launched in a different context altogether — waka-waka-waka, Oct 25 '15 at 00:00
@waka-waka-waka: working with a process that you have not started is a different issue. Ask a separate question. — jfs, Oct 25 '15 at 00:04

python subprocess multiple stdin.write and stdout.read

1 Answers1

Output

Linked