Python Subprocess reading from stdout blocks (reading output in real time)

Question

I am trying to interact with an application using subprocess. I've created the process using Popen but I am not able to access the output stream without blocking the whole thread. Writing to the inputstream however seem to work fine (tested it using communicate, however I may not be able to use this later as I need real time data).

I have already tried putting the buffer to 1 but it doesnt seem to work.

I have noticed that sometimes if the process terminates, the output is flushed. I do believe that this issue may be caused by the fact that no flushing occurs (and that on closing, all the data gets recived at the same time) but I am not sure.

C code:

#include <stdio.h>
int main()
{
    int testInteger;
    printf("Enter an integer: \n");
    scanf("%d", &testInteger);
    printf("Number = %d",testInteger);
    return 0;
}

Python code

import subprocess

p = subprocess.Popen("./a.out", stdout=subprocess.PIPE, stderr=subprocess.PIPE, stdin=subprocess.PIPE, universal_newlines=True, bufsize=1, close_fds=True)

print(p.stdout.read(1)) #This should read E but instead blocks the whole thread!

Answer 1

I have already tried putting the buffer to 1 but it doesnt seem to work.

The bufsize parameter specifies the buffering of the pipe, but the binary you're calling has its own streams buffering which is generally full buffering if the binary is not outputting to a terminal (it's line buffering if stdout is a term).

You can observe this if you change the communication channel to stderr (using fprintf). Or if you fflush(stdout) explicitely after the printf. Or if you explicitely change the buffering configuration using setbuf(3)/setvbuf(3) (warning: this is UB unless it's done immediately at program start).

If you do not wish to modify the C program you can also use stdbuf (very much GNU-specific) to customise the buffering of the wrapped binary, just replace "./a.out" by ['stdbuf', '-o0', 'a.out'] to run a.out with an unbuffered stdout.

Incidentally, that sort of mess is why you probably don't want to handroll scripting interactive programs, that's why pexpect exists.

Oh, and stdin generally has the same buffering as stdout, by default (so line-buffered when hooked to a terminal and fully buffered otherwise).

Answer 2

The pipe read() waits for the subprocess to terminate before returning the entire output, hence it blocks.

You can try using readline() as described in read subprocess stdout line by line.

Edit:

Your c program might need to fflush(stdout) after printf. If printf detects a pipe then it can choose to not flush even with output of \n.

See more at Does printf always flush the buffer on encountering a newline?.