subprocess readline hangs waiting for EOF

Question

I have a simple c++ program that I'm trying to execute through a python script. (I'm very new to writing scripts) and I'm having trouble reading output through the pipe. From what I've seen, it seems like readline() won't work without EOF, but I want to be able to read in the middle of the program and have the script respond to whats being outputted. Instead of reading output, it just hangs the python script:

#!/usr/bin/env python
import subprocess
def call_random_number():
    print "Running the random guesser"
    rng = subprocess.Popen("./randomNumber", stdin=subprocess.PIPE, stdout=subprocess.PIPE, shell=True)
    i = 50
    rng.stdin.write("%d\n" % i)
    output = rng.stdout.readline()
    output = rng.stdout.readline()

call_random_number()

and the c++ file, which generates a random number between one and 100, then checks the users guess until they guess correctly

#include<iostream>
#include<cstdlib>

int main(){
  std::cout<< "This program generates a random number from 1 to 100 and asks the user to enter guesses until they succuessfully guess the number.  It then tells the user how many guesses it took them\n";
  std::srand(std::time(NULL));
  int num = std::rand() % 100;
  int guessCount = 0;
  int guess = -1;
  std::cout << "Please enter a number:  ";
  std::cin >> guess;
  while(guess != num){
    if (guess > num){
        std::cout << "That guess is too high.  Please guess again:  ";
    } else {
        std::cout << "That guess is too low.  Please guess again:  ";
    }
    std::cin >> guess;
    guessCount++;
  }
  std::cout << "Congratulations!  You solved it in " << guessCount << " guesses!\n";
}

the eventual goal is to have the script solve the problem with a binary search, but for now I just want to be able to read a line without it being the end of the file

Answer 1

As @Ron Reiter pointed out, you can't use readline() because cout doesn't print newlines implicitly -- you either need std::endl or "\n" here.

For an interactive use, when you can't change the child program, pexpect module provides several convenience methods (and in general it solves for free: input/output directly from/to terminal (outside of stdin/stdout) and block-buffering issues):

#!/usr/bin/env python
import sys

if sys.version_info[:1] < (3,):
    from pexpect import spawn, EOF # $ pip install pexpect
else:
    from pexpect import spawnu as spawn, EOF # Python 3

child = spawn("./randomNumber") # run command
child.delaybeforesend = 0 
child.logfile_read = sys.stdout # print child output to stdout for debugging
child.expect("enter a number: ") # read the first prompt
lo, hi = 0, 100
while lo <= hi:
    mid = (lo + hi) // 2
    child.sendline(str(mid)) # send number
    index = child.expect([": ", EOF]) # read prompt
    if index == 0: # got prompt
        prompt = child.before
        if "too high" in prompt:
            hi = mid - 1 # guess > num
        elif "too low" in prompt:
            lo = mid + 1 # guess < num
    elif index == 1: # EOF
        assert "Congratulations" in child.before
        child.close()
        break
else:
    print('not found')
    child.terminate()
sys.exit(-child.signalstatus if child.signalstatus else child.exitstatus)

It works but it is a binary search therefore (traditionally) there could be bugs.

Here's a similar code that uses subprocess module for comparison:

#!/usr/bin/env python
from __future__ import print_function
import sys
from subprocess import Popen, PIPE

p = Popen("./randomNumber", stdin=PIPE, stdout=PIPE,
          bufsize=1, # line-buffering
          universal_newlines=True) # enable text mode
p.stdout.readline() # discard welcome message: "This program gener...

readchar = lambda: p.stdout.read(1)
def read_until(char):
    buf = []
    for c in iter(readchar, char):
        if not c: # EOF
            break
        buf.append(c)
    else: # no EOF
        buf.append(char)
    return ''.join(buf).strip()

prompt = read_until(':') # read 1st prompt
lo, hi = 0, 100
while lo <= hi:
    mid = (lo + hi) // 2
    print(prompt, mid)
    print(mid, file=p.stdin) # send number
    prompt = read_until(':') # read prompt
    if "Congratulations" in prompt:
        print(prompt)
        print(mid)
        break # found
    elif "too high" in prompt:
        hi = mid - 1 # guess > num
    elif "too low" in prompt:
        lo = mid + 1 # guess < num
else:
    print('not found')
    p.kill()
for pipe in [p.stdin, p.stdout]:
    try:
        pipe.close()
    except OSError:
        pass
sys.exit(p.wait())

Answer 2

I'm pretty sure adding newlines in your C++ program will cause the readlines to return.

Answer 3

You may have to explicitly closestdin, so the child process will stop hanging, which I think is what is happening with your code -- this can be verified by running top on a terminal and checking if randomnumber's status stays sleeping and if it is using 0% CPU after the expected time it would take to execute.

In short, if you add rng.stdin.close() right after the rng=subprocess(...) call, it may resume with no problem. Another option would be to do output=rng.communicate(stdin="%d\n" % i)and look at output[0]andoutput[1]who are stdout and stderr, respectively. You can find info oncommunicate here.