Python3 subprocess.Popen.readline is not live

Question

I try to run this script:

file = open("console-output.txt", "w")
task = subprocess.Popen(sys.executable + " \"main.py\"", shell=True, stdout=subprocess.PIPE, stderr=subprocess.STDOUT)
for line in iter(task.stdout.readline, ''):
  print("Got data")
  file.write(line)
  file.flush()
file.close()

It works fine and prints the program output to the console-output.txt file. However it outputs every text at once at the end of the program. I would like to have a live output to my file so that I can see the output of long-running programs. Am I doing anything wrong or is this a bug? I am on Ubuntu 17.10 with Python 3.6.3 64Bit.

It seems to me like task.stdout.readline is blocking till the program is completely finished.

Please outline what is different from my code? This is not a duplicate since IMHO my code should work. — fameman, Mar 09 '18 at 18:04
Are you sure that the `main.py` script is actually using line-buffered output, or is flushing its output on every line? If not, you'll have to wait for a block to fill up or for the subprocess to exit before the output becomes available to read in the parent process. — Daniel Pryden, Mar 09 '18 at 18:09
As an aside: There is absolutely no point (and a non-trivial security risk) in using a string argument and `shell=True` here. Just use `subprocess.Popen([sys.executable, "main.py"], ..)` and there's no need for a shell process at all. — Daniel Pryden, Mar 09 '18 at 18:10
Not a direct answer, but closely related: http://pexpect.readthedocs.io/en/latest/FAQ.html#whynotpipe — Daniel Pryden, Mar 09 '18 at 18:15
Possible duplicate of [Intercepting stdout of a subprocess while it is running](https://stackoverflow.com/questions/527197/intercepting-stdout-of-a-subprocess-while-it-is-running) — Davis Herring, Mar 09 '18 at 23:01

score 0 · Accepted Answer · answered Mar 10 '18 at 22:05

After a lot more research using different search terms, I found out that C (and therefore many interpreters and programs) detects whether the program output is a console or a pipe. It buffers every output, as soon as the buffer has enough empty space or is not flushed, if in pipe mode. To force an unbuffered behaviour, you just need to pass -u to the target python interpreter. If not using Python, you may want to try the stdbuf command pre-installed on almost all common linux platforms, available via the coreutils package in Mac OS X (you have to call gstdbuf instead of stdbuf ). After a lot of research, I found out that the only equivalent for stdbuf on linux could be the stdbuf.exe found in the git-scm for windows. However I did not test it yet.

Python3 subprocess.Popen.readline is not live

1 Answers1