How read in realtime an output process with python?

Question

I need to realise a python script who read the ouput of an other process in real time line by line. Obviously I've ever try to use "subprocess" to get the output process with stdout. But i can't get that output in real time, indeed every times python script return the output in several chunk few time after the process launch. For exemple i use this python script :

import subprocess as sub
import sys

proc = sub.Popen(["python", "bin/read_test.py"], 
                 stdout = sub.PIPE, 
                 bufsize = 1)

while True:
        if proc.poll() is None:
                line = proc.stdout.read()
                line = line.decode("utf-8")
                print(line, flush = True)
                sys.stdout.flush()
        else:
                proc.stdout.close()
                break

read_test.py script to be read :

from time import sleep

for i in range(5):
        print(i)
        sleep(1)

I've try a lot of methods with "readline()" with for loops but the issue still the same.

Moreover I don't want to use "communicate" because it's a blocking method.

Thanks for your help,

Possible duplicate of [read subprocess stdout line by line](http://stackoverflow.com/questions/2804543/read-subprocess-stdout-line-by-line) — pltrdy, Feb 20 '17 at 14:15

score 1 · Accepted Answer · answered Feb 20 '17 at 14:17

Problem is that you're trying to read stdout fully.

Since python sees that process is still running, it waits until process ends so process output is complete.

To do what you want you probably want to read the output line by line using line = proc.stdout.readline()

You probably have to change your loop to read line, and stop when process ends

proc = sub.Popen(["python", "bin/read_test.py"],
                 stdout = sub.PIPE,
                 bufsize = 1)

while True:
    line = proc.stdout.readline()
    if line:
        line = line.decode("utf-8")
        print(line)
        sys.stdout.flush()
    if proc.poll() is not None:  # process ends
        break

proc.wait()

Also that's not enough: you have to flush the output on the read_test.py side to make sure that the emitter actually sends the lines (when output is redirected, it's not sure). Example program:

import time,sys

for i in range(5):
    print(i)
    sys.stdout.flush()  # not flushing: you get all the lines at the end of the process
    time.sleep(1)

I've connected both programs, and got 1 sequenced output (1,2,3,4,5) each second

Ok, add the flush in the read_test.py have solved my issue, thank you. — Guillaume, Feb 20 '17 at 14:41

How read in realtime an output process with python?

1 Answers1