Get output from a program with the user input that program takes

Question

I'm trying to capture a string from the output of a subprocess and when the subprocess asks for user input, include the user input in the string, but I can't get stdout to work.

I got the string output from stdout using a while loop, but I don't know how to terminate it after reading the string.

I tried using subprocess.check_output, but then I can't see the prompts for user input.

import subprocess
import sys
child = subprocess.Popen(["java","findTheAverage"], stdout = subprocess.PIPE, stdin = subprocess.PIPE )
string = u""

while True:
    line = str(child.stdout.read(1))
    if line != '':
        string += line[2]
        print(string)
    else:
        break

print(string)
for line in sys.stdin:
    print(line)
    child.stdin.write(bytes(line, 'utf-8'))

---

EDIT:

With help and code from Alfe post I now have a string getting created from the subprocess programs output, and the users input to that program, but its jumbled about.

The string appears to first get The first letter of the output, then the user input, then the rest of the output.

Example of string muddling:

U2
3ser! please enter a double:U
4ser! please enter another double: U
5ser! please enter one final double: Your numbers were:
a = 2.0
b = 3.0
c = 4.0
average = 3.0

Is meant to be:

User! please enter a double:2
User! please enter another double: 3
User! please enter one final double: 4
Your numbers were:
a = 2.0
b = 3.0
c = 4.0
average = 3.0

Using the code:

import subprocess
import sys
import signal
import select


def signal_handler(signum, frame):
    raise Exception("Timed out!")


child = subprocess.Popen(["java","findTheAverage"], universal_newlines = True, stdout = subprocess.PIPE, stdin = subprocess.PIPE )
string = u""
stringbuf = ""

while True:
  print(child.poll())
  if child.poll() != None and not stringbuf:
    break
  signal.signal(signal.SIGALRM, signal_handler)
  signal.alarm(1)
  try:
    r, w, e = select.select([ child.stdout, sys.stdin ], [], [])
    if child.stdout in r:
        stringbuf = child.stdout.read(1)
        string += stringbuf
        print(stringbuf)
  except:
    print(string)
    print(stringbuf)
  if sys.stdin in r:
    typed = sys.stdin.read(1)
    child.stdin.write(typed)
    string += typed

---

FINAL EDIT:

Alright, I played around with it and got it working with this code:

import subprocess
import sys
import select
import fcntl
import os

# the string that we will return filled with tasty program output and user input #
string = ""

# the subprocess running the program #
child = subprocess.Popen(["java","findTheAverage"],bufsize = 0, universal_newlines = True, stdout = subprocess.PIPE, stdin = subprocess.PIPE )

# stuff to stop IO blocks in child.stdout and sys.stdin ## (I stole if from  http://stackoverflow.com/a/8980466/2674170)
fcntl.fcntl(child.stdout.fileno(), fcntl.F_SETFL, os.O_NONBLOCK)
fcntl.fcntl(sys.stdin.fileno(), fcntl.F_SETFL, os.O_NONBLOCK)

# this here in the unlikely event that the program has #
# finished by the time the main loop is first running  #
# because if that happened the loop would end without  #
# having added the programs output to the string!      #
progout = ""
typedbuf = "#"

### here we have the main loop, this friendly fellah is 
### going to read from the program and user, and tell
### each other what needs to be known
while True:

## stop when the program finishes and there is no more output
  if child.poll() != None and not progout:
    break

# read from 
  typed = ""

  while typedbuf:
    try:
      typedbuf = sys.stdin.read(1)
    except:
      break
    typed += typedbuf
    stringbuf  = "#"
  string += typed
  child.stdin.write(typed)

  progout = ""
  progoutbuf = "#"
  while progoutbuf:
    try:
      progoutbuf = child.stdout.read(1)
    except:
      typedbuf = "#"
      break 
    progout += progoutbuf
  if progout:
    print(progout)
  string += progout

# the final output string #
print( string)

Answer 1

You need select to read from more than one source at the same time (in your case stdin and the output of the child process).

import select

string = ''
while True:
  r, w, e = select.select([ child.stdout, sys.stdin ], [], [])
  if child.stdout in r:
    string += child.stdout.read()
  if sys.stdin in r:
    typed = sys.stdin.read()
    child.stdin.write(typed)
    string += typed

You will still need to find a proper breaking condition to leave that loop. But you probably get the idea already.

I want to give a warning at this point: Processes writing into pipes typically buffer until the latest possible moment; you might not expect this because when testing the same program from the command line (in a terminal) typically only lines get buffered. This is due to performance considerations. When writing to a terminal, typically a user expects to see the output as soon as possible. When writing to a pipe, typically a reading process is happy to be given larger chunks in order to sleep longer before they arrive.