Real Time updating executable subprocess console output in Python

Question

I have been reading this question, and this one and noted that the solutions proposed there did not do the trick I'm trying to pull off. The idea is similar to this question but I'm giving a better example here.

I need to build a GUI in Python 3.7 to monitor and control the execution of a legacy software written in C. Right now, I'm trying to simply call a test executable from Python and have it printed on a GUI, but for now it would suffice to print on the same console as python. The problem is that the executable takes very long to fully execute, but prints console messages in the meantime, I need these messages to be read promptly by my GUI.

Here is a working example:

in Python 3.7:

import sys
from threading import Thread
import time
import subprocess

class HandleTarget(Thread):
    def __init__(self):
        Thread.__init__(self)

    def run(self):
        subprocess.Popen(["hello.exe"], stdout=sys.stdout, bufsize=1)

class Printer(Thread):
    def __init__(self):
        Thread.__init__(self)

    def run(self):
        for i in range(1, 15):
            sys.stdout.write("Hi back ("+str(i)+")!\n")
            time.sleep(1.0)

thread_1 = HandleTarget()
thread_2 = Printer()

# Launching threads:
thread_1.start()
thread_2.start()

Now, the example executable ("hello.exe") can be build in C like this:

#include <stdio.h>
#include <time.h>

int main() {
    struct timespec time;
    double tic = 0;
    double toc = 0;

    for( int ii = 0; ii < 3 ; ii++){
        clock_gettime(CLOCK_MONOTONIC, &time);
        tic = time.tv_sec;
        toc = tic;

        while(toc-tic<3) {
        clock_gettime(CLOCK_MONOTONIC, &time);
        toc    = time.tv_sec;
        }
        printf("Hello #%d \n",ii);
    }
    return(0); 
}

Ideally, I need the messages "Hello" and "Hi back" to be interleaved. But if I run the python script above, I get:

Hi back (1)!
Hi back (2)!
Hi back (3)!
Hi back (4)!
Hi back (5)!
Hi back (6)!
Hi back (7)!
Hi back (8)!
Hi back (9)!
Hello #0 
Hello #1 
Hello #2 
Hi back (10)!
Hi back (11)!
Hi back (12)!
Hi back (13)!
Hi back (14)!

Apparently, the output from the executable is only printed when the execution is finished. This means that if the legacy executable was running, only upon being finished anything would show up.

Edit: I do not have hard real time constraints on this, if printing actually takes a few minutes, that's fine, the problem is that if a process needs to run for days, then every few hours (preferably minutes) the GUI needs to be updated with the console prints from the executable.

output from executable is printed when there is "\n". But Python doesn't run threads at the same time - it runs one of them for awhile, next it runs second of them for awhile, next it runs first for awhile, etc. Sometimes on thread may run longer than another and you can get all text from one thread at once. I would use `stdout=PIPE` to read line from subprocess (when line has "\n") and display it in GUI and then wait for next line from subprocess. — furas, Apr 04 '19 at 23:15
By inverted, do you mean, instead of the *"hi back"* it will print *"hello"*, and vice versa? — xilpex, Apr 07 '19 at 22:36
@Xilpex : I actually wrote "intervened", which means that the hello messages should not all be together, but rather there should be one *hi back* between two *hello*. — Mefitico, Apr 07 '19 at 23:34
@Massimo According to [Dictionary.com](https://www.dictionary.com/browse/intervene) Intervene "2.to occur or be between two things. 3 to occur or happen between other events or periods". But I agree "interleaved" would be clearer. — Mefitico, Jun 13 '19 at 12:05

gbajson · Accepted Answer · 2019-04-10T14:00:34.940

Consider reading Popen.stdout:

https://docs.python.org/3/library/subprocess.html?highlight=subprocess#subprocess.Popen.stdout

Here is the output of the updated program:

$ python test.py
Hi back (1)!
Hi back (2)!
Hi back (3)!
Hello #0
Hi back (4)!
Hi back (5)!
Hi back (6)!
Hello #1
Hi back (7)!
Hi back (8)!
Hi back (9)!
Hello #2
Hi back (10)!
Hi back (11)!
Hi back (12)!
Hi back (13)!
Hi back (14)!

The updated program:

import sys
from threading import Thread
import time
import subprocess

class HandleTarget(Thread):
    def __init__(self):
        Thread.__init__(self)

    def run(self):
        proc = subprocess.Popen(["hello.exe"], stdout=subprocess.PIPE)
        for line in proc.stdout:
            print(line.strip().decode('utf-8'))


class Printer(Thread):
    def __init__(self):
        Thread.__init__(self)

    def run(self):
        for i in range(1, 15):
            sys.stdout.write("Hi back ("+str(i)+")!\n")
            time.sleep(1.0)

thread_1 = HandleTarget()
thread_2 = Printer()

# Launching threads:
thread_1.start()
thread_2.start()

EDIT:

I also modified C program to flush stdout after printing, buffered stdout seems to be the root cause of the problem:

    printf("Hello #%d \n",ii);
    fflush(stdout);

On the bright side, this worked here as well. But as mentioned in the question, there is little I'm allowed to do with the legacy software, the hello function is meant to be a minimal working example. I might try to redefine `printf` to always perform an `fflush` and preload a custom lib, or something. Nonetheless, if no better answer comes about, you are eligible to the bounty. — Mefitico, Apr 10 '19 at 14:08
I checked that it also works to add `setvbuf(stdout, NULL, _IONBF, 0);` (prevent buffering altogether). This might be easier on my requirements, you might consider testing/adding to your answer. — Mefitico, Apr 10 '19 at 16:41
@Mefitico I recompiled C program in Linux and it prints "Hello #%d \n" every second. The same source compiled on Windows flushes buffer at once when program ends. Therefore I think it's rather a problem with the buffers in Windows terminal, than Python. — gbajson, Apr 11 '19 at 08:38

Real Time updating executable subprocess console output in Python

1 Answers1

EDIT: