subprocess: can't get live output if time between outputs is too big

Question

I want to start a python script (I will call it test.py) in another python script (run_python_script.py). For that I use the subprocess command like shown here: https://stackoverflow.com/a/18422264.

So that is run_python_script.py :

import subprocess
import sys
import io
import time


def runPython(filename, filename_log="log.txt"):

    with io.open(filename_log, 'wb') as writer, io.open(filename_log, 'rb', 1) as reader:
        process = subprocess.Popen("python {}".format(filename), stdout=writer)
        while process.poll() is None:
            sys.stdout.write(reader.read().decode("utf-8"))
            time.sleep(0.5)
        # Read the remaining
        sys.stdout.write(reader.read().decode("utf-8"))


runPython("test.py")

And this is test.py :

import time

sleep_time = 0.0001
start_time = time.time()

for i in range(10000):
    print(i)
    time.sleep(sleep_time)
print(time.time() - start_time)

In this setup the live output works, but if sleep_time (in test.py) is to big, for example sleep_time = 1. run_python_script.py only outputs after test.py is finished.
I replaced time.sleep(sleep_time) with other functions and every function which takes to long, breaks the live output.

Of corse test.py is just an example. I want to use other methods, but the result is the same.

Answer 1

To get live output from the child process you need to do some buffer flushing. I assume you're using Python 3, which supports the flush arg to the print function.

I also assume you want to pass text data from the child to the parent, but it's also easy to pass binary data: get rid of the universal_newlines=True and change the bufsize to zero (or some buffer size appropriate to your data).

yannick_test.py

from time import sleep, perf_counter

sleep_time = 0.5
start_time = perf_counter()

for i in range(10):
    print(i, flush=True)
    sleep(sleep_time)

print(perf_counter() - start_time)

run_python_script.py

from subprocess import Popen, PIPE

def run_python(filename):
    process = Popen(["python3", filename],
        universal_newlines=True, bufsize=1, stdout=PIPE)
    for data in process.stdout:
        print(data, end='', flush=True)

run_python("yannick_test.py")

typical output

0
1
2
3
4
5
6
7
8
9
5.0068185229974915

---

Although this works, it would be more efficient to structure your child scripts so that you can import them and call their functions, either directly, or using threads or multiprocessing if you need things to run simultaneously.