Python: How to get an output of subrocess everytime it's outputting and not on terminating?

Question

I have two scripts. (Code is listed below). First script is just printing numbers with a delay.

Second script is supposed to start the first one as a subprocess, then checking in the background (!) if the first script outputting a number, and printing it if so.

But what I get is that the second script printing all of the numbers at once when first one is terminated (after 20 seconds) and not one number at each moment. Output is like:

START
0
1
2
3
4
5
6
7
8
9
16:05:51
OVER

But what I expect is something like:

START
0
16:05:51
1
16:05:53
2
16:05:55
3
16:05:57
4
16:05:59
5
16:06:01
6
16:06:03
7
16:06:05
8
16:06:07
9
16:06:09
OVER

How can I solve this?

Code:

1.py:

import time
for i in range(10):
    print(i)
    sys.stdout.flush()
    time.sleep(2)

test.py:

from subprocess import Popen, PIPE, STDOUT
from time import strftime
import threading, sys

stream = Popen(['python', '1.py'], stdout=PIPE, stderr=STDOUT)
def a():
        while True:
            try:
                out = stream.communicate()[0].decode("utf-8")
                print(out, strftime("%H:%M:%S"))
                sys.stdout.flush()
            except ValueError:
                print('OVER')
                break
            except:
                raise
t = threading.Thread(target=a,)
t.start()
print('START')

loop over `stream.stdout` instead of using `communicate()` in this case — mata, Sep 25 '16 at 13:56
@mata Not working either. It waits for first script to finish and then output all at once. — Sergey Dylda, Sep 25 '16 at 14:00
You _also_ should check that you flush the output regularly in the first script. — mata, Sep 25 '16 at 14:04
@mata I did that already. Flushing after every print. But it's not helping. — Sergey Dylda, Sep 25 '16 at 14:12
Problem solved by setting `bufsize=1` in arguments of Popen object and runnung script in `-u` mode. Thanks to everyone! — Sergey Dylda, Sep 25 '16 at 14:44

score 0 · Accepted Answer · edited May 23 '17 at 12:19

Try calling

sys.stdout.flush()

after each printout!

You are basically writing into a buffer, and those writes don't become "flushed" out until the writer process is done! So, if you don't want to wait until the python interpreter (or the OS underneath) decides "it is time to flush" ... you have to enforce that step "manually"!

See that other answer for more background information!

EDIT: as this is not working with stream as you say: probably you have to other ways to invoke that second script. You could try using the pexpect module instead. That one exists for the "sole" purpose of "interacting" with stdout output!

Not working. It seems like `stream.communicate()[0]` waiting for subprocess to terminate. So that `print(out, strftime("%H:%M:%S"))` is not executing. — Sergey Dylda, Sep 25 '16 at 13:45

Python: How to get an output of subrocess everytime it's outputting and not on terminating?

1.py:

test.py:

1 Answers1