How to get live stream of output from a subprocess?

Question

I'm trying to extract output from a running Python subprocess. For simplicity, I created a C executable that just prints out 0, 1, 2, etc. every second and runs indefinitely.

The Python script I have, along with a few other variations of a similar idea, all seem to do nothing for 5 minutes and then print a chunk of 300 lines to the terminal all at once. Instead, I am trying to have it print one line every second. Using Python 3.5.

import subprocess
import os
import sys
from time import sleep 

def start_program():
    args = ['./test_program']
    p = subprocess.Popen(args, stdout=subprocess.PIPE, universal_newlines=True, bufsize=1)
    return p

def print_output(p):
    for line in p.stdout:
        print(line, end='')

def main():
    p = start_program()
    print_output(p)

main()

Answer 1

The problem is that your C program is detecting that the output isn't going to a TTY and is therefore buffering it. I assume your C program is something like this:

#include <stdio.h>
#include <unistd.h>

int main(void) {
    for(int i = 0; i < 300; ++i) {
        printf("%d\n", i);
        sleep(1);
    }
}

There's two ways you can fix it:

1. Edit your C program to do fflush(stdout); each iteration of the loop or setvbuf to disable buffering at the beginning
2. Edit your Python program to wrap your C program in stdbuf, like this: args = ['stdbuf', '-oL', './test_program']

Answer 2

The base problem was described by @Joseph Sible-Reinstate Monica, but you may also face temporary blocking issue. Using process.stdout.readline() or for line in process.stdout: makes the script wait for a new line in process.stdout.

You can receive and process data in real time without waiting by analyzing the availability of data before reading it.

Main file:

import subprocess
from time import sleep

process = subprocess.Popen(('python', 'subTest.py'), stdout=subprocess.PIPE)

while True:
    sleep(0.7) #do smth
    bytes_number = process.stdout.seek(0, 2)
    if bytes_number:
        print('number of new bytes: ', bytes_number)
        process.stdout.seek(0, 0)
        print(process.stdout.read(bytes_number))
    if process.poll() is not None and not bytes_number:
        break

print('done!')

If you run another python script instead of C-executable you also need to use stdout.flush() (analog fflush(stdout) in Python).

Subprocess file:

from sys import stdout
from time import sleep

for i in range(10):
    sleep(0.2) #do smth
    stdout.write(str(i) + '/')
    stdout.flush()