Get print() realtime output with subprocess

Question

I want to execute a Python file from another Python file and show all print() outputs and error outputs without waiting (realtime).

The simplified version of my code is as follows and I would like to show "start" and an error message without waiting for "end" (the end of the script).

def main():
    # Function that takes a long time (in my actual code)
    x += 1 # this raises an error

if __name__ == "main":
    print("start")
    main()
    print("end")

I also have run.py:

import subprocess

def run():
    subprocess.run(["python", "main.py"])


if __name__ == '__main__':
    run()

I tried this blog post and several other similar answers on stackoverflow, but none of them worked, so I decided to put my original code here, which is above.

try using check_output() https://stackoverflow.com/a/8235171/546375 — Alex M, Aug 19 '18 at 13:21
What difficulties did you have implementing the blog post you linked? stdout is buffered so you might want to do `print('start', flush=True)` to force output so the listening process has something to retrieve... — Jon Clements, Aug 19 '18 at 13:23
I do not think subprocess has a `run()` method, it should be `call()`, right? — Ketan Mukadam, Aug 19 '18 at 14:06
@KetanMukadam: [It does in Python 3.5+.](https://docs.python.org/3/library/subprocess.html#subprocess.run) — jwodder, Aug 19 '18 at 14:35

score 1 · Answer 1 · answered Aug 19 '18 at 14:04

1

Is this line a mistake?

if __name__ == "main":

The symbol is __main__ set by interpreter and not main. It is possible that because of this typo error no code is running from main script. Try first executing the main script directly on command shell.

answered Aug 19 '18 at 14:04

Ketan Mukadam

789
3
7

score 1 · Accepted Answer · answered Aug 19 '18 at 14:31

The following seems to work for me (on Windows). It uses subprocess.Popen() to execute the other script because this gives more control over what goes on. It turns buffering off to eliminate any delays that might cause, plus it redirects stderr to stdout to so all output can be retrieved from a single source.. Also note it also includes the correction @Ketan Mukadam mentions is his answer tregarding the value of __name__ in your first script.

main_script.py:

def main():
    # Function that takes a long time (in my actual code)
    x += 1 # this raises an error

if __name__ == '__main__':
    print("start")
    main()
    print("end")

run.py:

import subprocess
import sys

def run():
    kwargs = dict(bufsize=0,  # No buffering.
                  stdout=subprocess.PIPE,
                  stderr=subprocess.STDOUT,  # Redirect stderr to stdout.
                  universal_newlines=True)
    args = [sys.executable, 'main_script.py']

    with subprocess.Popen(args, **kwargs).stdout as output:
        for line in output:
            print(line, end='')  # Process the output...

if __name__ == '__main__':
    run()

Output from executing run.py:

start
Traceback (most recent call last):
  File "main_script.py", line 10, in <module>
    main()
  File "main_script.py", line 6, in main
    x += 1 # this raises an error
UnboundLocalError: local variable 'x' referenced before assignment

Get print() realtime output with subprocess

2 Answers2