Calling python script with subprocess.Popen and flushing the data

Question

Ok so i've seen dozen of threads like that , but none of them gives a complete answer and everything i tried so far foes not work for me.

1) Script that constantly outputs some data and flusheshs it:

import time
import sys

if __name__ == '__main__':
    for i in range(5):
        print i,
        sys.stdout.flush()
        time.sleep(1)

2) Script that calls first script with Popen and should be printing numbers one by one but for some reason does not, and prints them alltogether at once :

import sys
import subprocess

if __name__ == '__main__':
    process = subprocess.Popen(['python', 'flush.py'], stdout = subprocess.PIPE )
    for line in iter(process.stdout.readline, ''):
        print line,
        sys.stdout.flush()

First thing i am a little bit confused is in the first script is that if you remove the flush it returns output in one line alltogether O_O... I am pretty sure it is because of time.sleep but still kind of expected it return like a standart output constantly returning values 0,1,2,3,4 but not all together, ofcourse flush resolves it , but just strange, at least for me ...

The main problem: Is that second script does not return number one by one , but returns all in one output at once..... What i need is to see numbers popping one by one...

I read somewhere that it does not return EOF which Popen waits to close the pipe , thats why it runs like to the end .....

So what do i do or try next ? Thanks in advance.

Probably because your subprocess is not printing a newline after each number. `readline` reads until a newline (or end of file). See if my answer here helps: http://stackoverflow.com/questions/12255802/having-difficulty-capturing-output-of-a-subprocess-that-does-in-place-status-upd/12256356#12256356 — Warren Weckesser, Jan 22 '13 at 16:28
Well, basically i found solution to my question . Here is the code that works although i don't really grasp the theory behind it [link](http://chase-seibert.github.com/blog/2012/11/16/python-subprocess-asynchronous-read-stdout.html). Why do you have execute it in thread, somewhere i read not to block Main thread, but then again how it helps , if you still get the whole text together only in thread, that makes no sense to me , because i dont understand how it works i guess. Then the thing about setting file descriptor to non blocking that confuses me a lot too. — Viktor, Jan 23 '13 at 15:54
@Viktor: [the link that you provided](http://chase-seibert.github.com/blog/2012/11/16/python-subprocess-asynchronous-read-stdout.html) is unnecessary complicated. It works because it doesn't use `readline()` (that won't work for non-blocking pipes anyway) and therefore it doesn't wait for a newline that your child script never produces. Here's a [simpler solution that should be enough for line-oriented output in many cases](http://stackoverflow.com/a/17698359/4279) — jfs, Dec 13 '13 at 21:24

score 13 · Answer 1 · edited May 23 '17 at 11:46

As @Warren Weckesser's comment says, your problem is unrelated to buffering issues.

.readline() in the parent process won't return until it reads a newline or reaches EOF. Your child process doesn't print any newlines at all so your parent process doesn't print anything until the child process ends.

The minimal fix is just to remove comma at the end of print i, in the child script.

This also works:

#!/usr/bin/env python
import sys
from subprocess import Popen, PIPE

p = Popen([sys.executable or 'python',
           '-u', # unbuffer stdout (or make it line-buffered on Python 3)
           '-c',
           """
import time

for i in range(5):
    print(i) # <-- no comma i.e., each number is on its own line
    time.sleep(1)
"""], stdout=PIPE, bufsize=1)
for line in iter(p.stdout.readline, b''):
    print(int(line)**2)

Example:

 $ python parent.py
 0
 1
 4
 9
 16

The numbers are printed every seconds without waiting for the child process to end.

If you don't want to change the child script then you should use readline() that stops at whitespace instead of a newline character e.g.:

#!/usr/bin/env python
import sys
from subprocess import Popen, PIPE

p = Popen(['python2', 'child.py'], stdout=PIPE, bufsize=0)
for token in generate_tokens(p.stdout):
    print(int(token))

where generate_tokens() yields whitespace-separated tokens:

def generate_tokens(pipe):
    buf = []
    while True:
        b = pipe.read(1) # read one byte
        if not b: # EOF
            pipe.close()
            if buf:
                yield b''.join(buf)
            return
        elif not b.isspace(): # grow token
            buf.append(b)
        elif buf: # full token read
            yield b''.join(buf)
            buf = []

It also prints integers as soon as they are printed by the child.

Calling python script with subprocess.Popen and flushing the data

1 Answers1

Linked