Real-time stdout read from subprocess works only when running from PyCharm and not Terminal

Question

What the application does: runs a subprocess and displays the stdout in real-time to a Tkinter textbox widget.

This works perfectly when I run the application from PyCharm.

When I run the application from terminal ./application.py it doesn't display in real-time, but instead will display it all after the process has finished.

Some details:

I have a subprocess running (the subprocess prints out "SPAM" every 1 second for 10 seconds):

process = subprocess.Popen(<some file path>, stdout=subprocess.PIPE, universal_newlines=True)

I am printing the stdout to a Tkinter textbox widget:

for stdout_line in iter(process.stdout.readline, ""):
     self.app.pm_textbox.see(tk.END)
     self.app.pm_textbox.insert(tk.INSERT, stdout_line)

So my question is what could possibly cause running from terminal and PyCharm to display stdout data differently?

It's inheriting the current `stdin` and `stderr`. Probably in PyCharm those are pipes. I'm not sure, though. I think PyCharm works a lot of magic for debugging, and I don't have it installed to test what it does. When run from an interactive shell, `stdin` and `stderr` should be for the terminal. Try overriding them to `subprocess.DEVNULL`, so they're consistently the same in both cases. — Eryk Sun, Apr 27 '17 at 01:02
Are you developing on Linux or some other Unix-like OS that has the `stdbuf` command? If so, try changing the command to `['stdbuf', '-oL', filepath]` to override the default to line buffering. If that works, I wonder what PyCharm is doing. — Eryk Sun, Apr 27 '17 at 17:25
@eryksun I am developing on Ubuntu 16.04. Adding this command didn't seem to change anything - PyCharm is still working and launching the application from the terminal does not. — Cov, Apr 28 '17 at 08:39
The symptoms here look like it's fully buffering output before writing it to the pipe. Typical buffer sizes are 4 KiB or 8 KiB, so if you don't have that much data, the buffer only gets flushed to the pipe when the program exits. `stdbuf` won't always work. It only changes the default behavior, but some programs override this. — Eryk Sun, Apr 28 '17 at 11:50

score 0 · Answer 1 · answered Jun 26 '19 at 10:45

I know this is an old thread, but still -

I had the same issue as you did. I asked a question that seemed to help me get in the right direction as to working with real time output. My issue was I needed to differentiate output that resulted of a CR and simulate that behavior in my program. But then I encountered your problem as well - I worked for 3 days trying to find a solution as to why this happens.

After reading this and having no luck with what they did there, I found this thread without answers but one comment that after a little modification, finally helped me.

What finally worked for me, is this:

#                        This seems to handle    This seems to handle
#                        the terminal output     the PyCharm output
proc = subprocess.Popen(["stdbuf", "-oL"] + cmd, stdout=subprocess.PIPE)

out = io.open(proc.stdout.fileno(), mode='r', encoding="utf-8", newline='')
should_cr = False
for line in out:
    if should_cr is True:
        sys.stdout.write("\r")
        sys.stdout.flush()

    if line.endswith(os.linesep):
        should_cr = False
        sys.stdout.write(line)
        sys.stdout.flush()

    elif line.endswith("\r"):
        should_cr = True
        line = line.rstrip()
        sys.stdout.write(line)
        sys.stdout.flush()

If you wouldn't regard CR output then it is more simple of course:

#                        This seems to handle    This seems to handle
#                        the terminal output     the PyCharm output
proc = subprocess.Popen(["stdbuf", "-oL"] + cmd, stdout=subprocess.PIPE)
for line in iter(proc.stdout.readline, ''):
    print(line)

Notice this works for me in Python 2.7

score 0 · Answer 2 · answered Nov 10 '22 at 08:16

0

Adding the following worked for me. Just put this before executing "subprocess.Popen()"

os.environ["PYTHONUNBUFFERED"] = "1"

answered Nov 10 '22 at 08:16

Алексей Новиков

23
4

Real-time stdout read from subprocess works only when running from PyCharm and not Terminal

2 Answers2