Live output from Python subprocess that calls Python script

Question

There's quite a bit of posts related to collecting live output from a process that was launched using Python's subprocess module. When I attempt these solutions between my two test scripts, one being a (ba)sh script and the other being a Python script, the Python script fails to have its output read live. Instead when the Python script is ran by subprocess it ends up waiting until the process has completed to flush it to PIPE. The constraints I'm bounded by is that I do want a way to retrieve live output from subprocess for the Python script.

Tested on Ubuntu 20.04 & Windows, Shell script ran on Ubuntu 20.04.

Calling code:

import shlex
import subprocess

# invoke process
process = subprocess.Popen('python test.py',shell=True,stdout=subprocess.PIPE) #Shell true/false results in "not live" output

# Poll process.stdout to show stdout live
while True:
  output = process.stdout.readline() # <-- Hangs here on calling test.py, doesn't hang on test.sh
  if process.poll() is not None:
    break
  if output:
    print(output.strip())
rc = process.poll()

test.py <-- Waits until it has completed to print out entire output

import time

for x in range(10):
   print(x)
   time.sleep(1)

test.sh <-- Prints out live in Python script

#!/bin/bash

for i in $(seq 1 5); do
   echo "iteration" $i
   sleep 1
done

https://stackoverflow.com/questions/107705/disable-output-buffering — stochastic13, Jun 14 '21 at 14:34
Gave this one up because I didn't know about the -u switch. Updating issue to reflect further what I'll be running into. — Kyle, Jun 14 '21 at 15:17

score 2 · Answer 1 · answered Jun 14 '21 at 16:44

@stochastic13 Provided a very useful link where the -u switch and PYTHONUNBUFFERED variable being set would work. For my needs, I used PYTHONUNBUFFERED which solved my issue entirely. The Python test script actually executes another Python script to run, which I needed the output on. Despite -u helping for the first script, it wouldn't help for the second as I wouldn't have direct access to said script to add the argument. Instead I went with the environment variable, solution below:

def run_command(command):
   os.environ['PYTHONUNBUFFERED'] = '1'
   process = Popen(command, shell=False, stdout=PIPE, env=os.environ) # Shell doesn't quite matter for this issue
   while True:
      output = process.stdout.readline()
      if process.poll() is not None:
         break
      if output:
         print(output)
   rc = process.poll()
   return rc

Above the code passes PYTHONUNBUFFERED and sets it to the environment, any spawned process in subprocess with this environment set will inherit PYTHONUNBUFFERED.

Test Script

import subprocess

from io import TextIOWrapper, TextIOBase, StringIO
from subprocess import PIPE, Popen, call
from tempfile import TemporaryFile

from sarge import run, Capture

# process = Popen('python test2.py', shell=False)
# while True:
#    if process.poll() is not None:
#       break
# rc = process.poll()

subprocess.call('python test2.py')

Test Script 2

import time
import os

print(list(os.environ.keys()))

for x in range(10):
   print('test2', x)
   time.sleep(1)

The output is a live capture of stdout from any Python process, not just after completion.

...
b'test2 0\r\n'
b'test2 1\r\n'
b'test2 2\r\n'
b'test2 3\r\n'
...
0

Live output from Python subprocess that calls Python script

1 Answers1