3

To emphasize, the problem is real time read instead of non-blocking read. It has been asked before, e.g. subprocess.Popen.stdout - reading stdout in real-time (again). But no satisfactory solution has been proposed.

As an example, the following code tries to simulate the python shell. 

import subprocess

p = subprocess.Popen(['python'], stdin=subprocess.PIPE, stdout=subprocess.PIPE)

while True:
    line = input('>>> ')
    p.stdin.write(line.encode())
    print('>>> ', p.stdout.read().decode())

However, it would be blocked when reading from p.stdout. After searching around, I found the following two possible soutions.

  1. using fctrl and O_NONBLOCK
  2. using thread and queue

Whereas the 1st soution may work and only work on linux, the 2nd soution just turn blocking read to non-blocking read, i.e. I cannot get real time output of the subprocess. For example, if I input 'print("hello")', I will get nothing from p.stdout using 2nd solution.

Perhaps, someone would suggest p.communite. Unfortunately, it is not suitable in this case, since it would close stdin as described here.

So, is there any solutions for Windows?

Edited: Even if -u is turned on and p.stdout.read is replaced with p.stdout.readline, the problem still exists.

import subprocess

p = subprocess.Popen(['python', '-u'], stdin=subprocess.PIPE, stdout=subprocess.PIPE)

while True:
    line = input('>>> ')
    p.stdin.write(line.encode())
    p.stdin.flush()
    print('>>> ', p.stdout.readline().decode())

Solution: The following is the final code based on J.F. Sebastian's answer and comments.

from subprocess import Popen, PIPE, STDOUT

with Popen(
        ['python', '-i', '-q'],
        stdin=PIPE, stdout=PIPE, stderr=STDOUT,
        bufsize=0
    ) as process:
    while True:
        line = input('>>> ')
        if not line:
            break
        process.stdin.write((line+'\n').encode())
        print(process.stdout.readline().decode(), end='')

It should be noted that the program would hang when the command triggers no output.

Community
  • 1
  • 1
cqdjyy01234
  • 1,180
  • 10
  • 20
  • "when reading from stdout" what? You can't read from stdout. – John Zwinck Apr 27 '16 at 03:42
  • 1
    the first sentence in the answer to [the question you've linked](http://stackoverflow.com/q/3140189/4279) describes the problem: "Trying to write to and read from pipes to a sub-process is tricky **because of the default _buffering_ going on in both directions.**" I have no idea how to force a child process to disable its internal stdout buffering in the general case on Windows—it is easy for `python` executable; just pass `-u` option and perhaps, `-i` to enable interactive mode (if you need to run some other command then read its docs). – jfs Apr 27 '16 at 15:35
  • unrelated: `p.stdout.read()` reads until EOF i.e., it is likely won't return until the child process is dead. – jfs Apr 27 '16 at 15:40
  • @J.F.Sebastian It seems so. Curiously, with `-u` and `p.stdout.readline()`, it doesn't work neither. – cqdjyy01234 Apr 28 '16 at 01:06
  • @cqdjyy01234 unless you are running a script; `-u` is not enough. To enable interactive mode, pass `-i` option, as I said above. Also, call `p.stdin.flush()` (to empty stdin buffer in the parent). – jfs Apr 28 '16 at 01:12
  • @J.F.Sebastian I think there exists misunderstandings. As edited, `-u` is passed to the subprocess to disable internal buffer, and `readline` is used to avoid the EOF problem. Moreover, `p.stdin.flush()` would not change anything. – cqdjyy01234 Apr 28 '16 at 01:32
  • @J.F.Sebastian Additionally, calling the script with `-u -i` doesn't work. – cqdjyy01234 Apr 28 '16 at 01:34
  • use either `python -u script.py` or `python -i` (no script file). There are *multiple* buffets!!! `p.stdin.flush()` is necessary (`bufsize=-1` by default) otherwise the child won't see you command (it will be inside the parent buffer) – jfs Apr 28 '16 at 01:36
  • @J.F.Sebastian OK. `-u`, `-i` or `-u -i` don't work. – cqdjyy01234 Apr 28 '16 at 01:38
  • @J.F.Sebastian Anyway, I cannot make it work. I have added `p.stdin.flush` to the script, named test.py. It is called using `python -u test.py` in the console. And after inputing, e.g. `print('hello\n')`, the console gets stuck without any output. – cqdjyy01234 Apr 28 '16 at 01:57
  • I don't see the script name in your code. If there is no script; use `-i`, to enable the interactive mode. The code should look like: `Popen(['python', '-i'], ...)`. Also, it seems a newline is missing at the of the command. – jfs Apr 28 '16 at 02:04
  • @J.F.Sebastian Of course, you could not see the script name. The sample code is saved in a file, e.g. test.py. Then it is used as the argument of the python command. – cqdjyy01234 Apr 28 '16 at 02:08
  • Look at the code in your question, remove `-u`, add `-i` – jfs Apr 28 '16 at 02:09
  • @J.F.Sebastian Although I didn't edit the question, I have tested before the response. `Popen(['python', '-i'], ...)` doesn't work. – cqdjyy01234 Apr 28 '16 at 02:11
  • don't put the solution into the question. [Post it as your own answer instead](http://stackoverflow.com/help/self-answer). btw, the solution may fail if the input does not produce *exactly* one line e.g., if it returns 0, 2, 3, ... lines. You could fix it by passing `stderr=STDOUT` and reading until the prompt (`>>>`) occurs—though I don't see why do you want to run `python` as a subprocess from within `python`—it might be easier to use `multiprocessing` or if you need something more complex, see the architecture of something like `jupyter`, `celery`. – jfs Apr 30 '16 at 17:15
  • Just want to emphasize that the comment of @jfs really work for me!!!! (the usage of '-u' in Windows when calling the python command). – Lukasavicus Apr 17 '20 at 14:37

1 Answers1

0

Here's a complete working example that uses a subprocess interactively:

#!/usr/bin/env python3
import sys
from subprocess import Popen, PIPE, DEVNULL

with Popen([sys.executable, '-i'], stdin=PIPE, stdout=PIPE, stderr=DEVNULL,
           universal_newlines=True) as process:
    for i in range(10):
        print("{}**2".format(i), file=process.stdin, flush=True)
        square = process.stdout.readline()
        print(square, end='')

Here's another example: how to run [sys.executable, '-u', 'test.py'] interactively.

Community
  • 1
  • 1
jfs
  • 399,953
  • 195
  • 994
  • 1,670
  • As you commented, there are 3 problems in my code: 1. `python` should be started with `-i`; 2. `input` would strip the newline, so I have to manually add a newline when passing the command to `p.stdin`; 3. `p.stdin` must be flushed, or else `bufsize` should be set to `0`. Thank you! – cqdjyy01234 Apr 28 '16 at 02:49