Issues intercepting subprocess output in real time

Question

I've spent about 6 hours on stack overflow, rewriting my python code and trying to get this to work. It just doesn't tho. No matter what I do.

The goal: Getting the output of a subprocess to appear in real time in a tkinter text box.

The issue: I can't figure out how to make the Popen work in real time. It seems to hang until the process is complete. (Run on its own, the process works completely as expected, so it's just this thing that has the error)

Relevant code:

import os
import tkinter
import tkinter.ttk as tk
import subprocess

class Application (tk.Frame):
    process = 0
    def __init__ (self, master=None):
        tk.Frame.__init__(self, master)
        self.grid()
        self.createWidgets()

    def createWidgets (self):
        self.quitButton = tk.Button(self, text='Quit', command=self.quit)
        self.quitButton.grid()
        self.console = tkinter.Text(self)
        self.console.config(state=tkinter.DISABLED)
        self.console.grid()

    def startProcess (self):
        dir = "C:/folder/"
        self.process = subprocess.Popen([ "python", "-u", dir + "start.py" ], stdout=subprocess.PIPE, stdin=subprocess.PIPE, stderr=subprocess.PIPE, cwd=dir)
        self.updateLines()

    def updateLines (self):
        self.console.config(state=tkinter.NORMAL)
        while True:
            line = self.process.stdout.readline().decode().rstrip()
            if line == '' and self.process.poll() != None:
                break
            else: 
                self.console.insert(tkinter.END, line + "\n")
        self.console.config(state=tkinter.DISABLED)
        self.after(1, self.updateLines)

app = Application()
app.startProcess()
app.mainloop()

Also, feel free to destroy my code if it's written poorly. This is my first python project, I don't expect to be any good at the language yet.

Answer 1

The problem here is that process.stdout.readline() will block until a full line is available. This means the condition line == '' will never be met until the process exits. You have two options around this.

First you can set stdout to non-blocking and manage a buffer yourself. It would look something like this. EDIT: As Terry Jan Reedy pointed out this is a Unix only solution. The second alternative should be preferred.

import fcntl
...

    def startProcess(self):
        self.process = subprocess.Popen(['./subtest.sh'],
            stdout=subprocess.PIPE,
            stdin=subprocess.PIPE,
            stderr=subprocess.PIPE,
            bufsize=0) # prevent any unnecessary buffering

        # set stdout to non-blocking
        fd = self.process.stdout.fileno()
        fl = fcntl.fcntl(fd, fcntl.F_GETFL)
        fcntl.fcntl(fd, fcntl.F_SETFL, fl | os.O_NONBLOCK)

        # schedule updatelines
        self.after(100, self.updateLines)

    def updateLines(self):
        # read stdout as much as we can
        line = ''
        while True:
            buff = self.process.stdout.read(1024)
            if buff:
                buff += line.decode()
            else:
                break

        self.console.config(state=tkinter.NORMAL)
        self.console.insert(tkinter.END, line)
        self.console.config(state=tkinter.DISABLED)

        # schedule callback
        if self.process.poll() is None:
            self.after(100, self.updateLines)

The second alternative is to have a separate thread read the lines into a queue. Then have updatelines pop from the queue. It would look something like this

from threading import Thread
from queue import Queue, Empty

def readlines(process, queue):
    while process.poll() is None:
        queue.put(process.stdout.readline())
...

    def startProcess(self):
        self.process = subprocess.Popen(['./subtest.sh'],
            stdout=subprocess.PIPE,
            stdin=subprocess.PIPE,
            stderr=subprocess.PIPE)

        self.queue = Queue()
        self.thread = Thread(target=readlines, args=(self.process, self.queue))
        self.thread.start()

        self.after(100, self.updateLines)

    def updateLines(self):
        try:
            line = self.queue.get(False) # False for non-blocking, raises Empty if empty
            self.console.config(state=tkinter.NORMAL)
            self.console.insert(tkinter.END, line)
            self.console.config(state=tkinter.DISABLED)
        except Empty:
            pass

        if self.process.poll() is None:
            self.after(100, self.updateLines)

The threading route is probably safer. I'm not positive that setting stdout to non-blocking will work on all platforms.