Monitor continual output with no newlines from one python script by another script with subprocess module

Question

When I tried to monitor the output of one python script from another none of the standard methods I could find were working. The things complicating this case is the first script need to continue running in parrallel and I need to monitor its output live. Also, the script does not have newlines - it just updates the current one.

Here's an MWE:

Script 1

import time
i=0
print("Starting")
while True:
    time.sleep(1)
    print("Running {: 5d}".format(i), end="\r")
    i+=1
print()

Run from the shell, this outputs:

Starting
Running     3

(where the second line updates once per second)

And my initial unsuccessful attempts to monitor it:

a = subprocess.Popen(["python3", "noNewLine1.py"], stdout=subprocess.PIPE, stderr=subprocess.PIPE)
while True :
    time.sleep(1)
    # b = a.stdout.readline() # Doesn't work - hangs
    # b = a.stdout.read(1)    # also hangs
    b = a.communicate(timeout=1) # Times out, no return
    print(b)

score 0 · Answer 1 · answered Mar 15 '23 at 14:49

Apparently python detects whether it is running in an interactive environment and changes its behavior accordingly. In this case, after running the script with subprocess.Popen it begins buffering output - but never (?) actually sends it to stdout. There's nothing that you can do to force it either (I thought a.stdout.flush() would be for that purpose, but no). So you have to tell python not to buffer output when you make the call by setting the PYTHONUNBUFFERED env variable.

Second problem is no newlines or end of file characters, so none of the nice read/communicate functions work. You have to read byte by byte, and decode the byes if you want to print out the same thing again. So here's an example of that mirrors the output from the first script:

import os
import subprocess
import time

os.environ['PYTHONUNBUFFERED'] = '1'
a = subprocess.Popen(["python3", "noNewLine1.py"], stdout=subprocess.PIPE, stderr=subprocess.PIPE, env=os.environ)

Third problem is that stdout.read(1) and even stdout.peek() block, so if you want to keep doing stuff while waiting if no output is ready, you have to implement some sort of threading solution. There's an adequate discussion here: A non-blocking read on a subprocess.PIPE in Python

while True:
    time.sleep(.1)
    b = a.stdout.read(1)
    print(b.decode(), end='')

Monitor continual output with no newlines from one python script by another script with subprocess module

1 Answers1