1

I'm using python to run some test cases for a C++ app I'm working on. When I run it in a terminal I see regular output right up until an assert(). With the following python code wrapping the call I miss a pile of stdout before the assert (yes, flush is not called, but why the difference between python and bash).

import subprocess as sp
cmd = ["myapp"]
proc = sp.Popen(cmd, stdout=sp.PIPE)

# runs in a thread while the main thread calls proc.wait(timeout)
for line in iter(proc.stdout.readline, b''):
    print(line.decode('ascii'))

The only fix I've found is to add cmd = ['stdbuf', '-o0'] + cmd (catching stdout...). Is there a python way to do whatever magic stdbuf does?

I noted that os.fdopen(proc.stdout.fileno(), 'r', 0) (without the b) produced ValueError: can't have unbuffered text I/O, which makes me wonder if using universal_newlines=True/text=True implicitly enables buffering and maybe even readline() is forced into it, but I'd have hoped that would happen within python and not in the pipe that gets destroyed when the subprocess crashes.

jozxyqk
  • 16,424
  • 12
  • 91
  • 180
  • When writing to a pipe, output is buffered by default. If the program crashes, the last buffer won't be flushed. – Barmar Aug 05 '22 at 21:58
  • You can do what `stdbuf` does by using the [`pty`](https://docs.python.org/3/library/pty.html) module. – Barmar Aug 05 '22 at 21:59
  • Thanks, @Barmar. Is it possible not to use a pipe? e.g. reading directly to a `StringIO`. What did you have in mind for `pty`? based on the [linked dupe](https://stackoverflow.com/questions/57147141/force-a-3rd-party-program-to-flush-its-output-when-called-through-subprocess), is it that `assert()` flushes only if stdout is a tty and I should use `pty.openpty()` with `Popen()`? – jozxyqk Aug 05 '22 at 23:06
  • The problems isn't the reading, it's the writing in the subprocess. You can't make it write directly into a string, it's writing to its stdout stream. The pty is the way you do it without a pipe. – Barmar Aug 05 '22 at 23:09
  • I see, so `Popen` necessarily creates a pipe. I guess it's in the name :P, but `stdout=sp.PIPE` implies it's configurable. Thanks! – jozxyqk Aug 05 '22 at 23:18
  • It defaults to a pipe, but you can override that. So you can create the pty and use that instead, as shown in the dup. – Barmar Aug 05 '22 at 23:20

0 Answers0