How to use pipes in python without blocking?

Question

In python, I want to create a subprocess and read and write data to its stdio. Lets say I have the following C program that just writes its input to its output.

#include <stdio.h>
int main() {
    char c;
    for(;;) {
        scanf("%c", &c);
        printf("%c", c);
    }
}

In python I should be able to use this using the subprocess module. Something like this:

from subprocess import *
pipe = Popen("thing", stdin=PIPE, stdout=PIPE)
pipe.stdin.write("blah blah blah")
text = pipe.stdout.read(4) # text should == "blah"

However in this case the call to read blocks indefinitely. How can I do what I'm trying to achieve?

Answer 1

stdout is line-buffered when writing to a terminal, but fully buffered when writing to a pipe so your output isn't being seen immediately.

To flush the buffer, call fflush(stdout); after each printf(). See also this question which is the same except that your subprocess is written in C, and this question which references stdin/stdout behaviour as defined in C99.

Answer 2

I found the pexpect module which does exactly what I need.