Python subprocess stdout iterator always blocks

Question

I want to capture the output while printing it, but I'm blocking forever without reading even a single line. What's going on? I'm using Python2.

Generator script:

#!/usr/bin/env python2.7

import random
import time

while True:
    print(random.random())
    time.sleep(1)

Sample generator output:

$ ./generator.py 
0.334835137212
0.896609571236
0.833267988558
0.55456332113
^CTraceback (most recent call last):

Reader script:

import subprocess

cmd = ['./generator.py']
p = subprocess.Popen(cmd, stdout=subprocess.PIPE)

for line in p.stdout:
    print(line)
    print('Looping')

p.wait()

I've also tried:

import subprocess
import sys

cmd = ['./generator.py']
p = subprocess.Popen(cmd, stdout=subprocess.PIPE)

while True:
    line = p.stdout.readline()
    print(line)

    print('Looping')

p.wait()

...and:

import sys
import subprocess
import select
import time

cmd = ['./generator.py']
p = subprocess.Popen(cmd, stdout=subprocess.PIPE)

s = select.poll()
s.register(p.stdout, select.POLLIN)

while True:
    if s.poll(1):
        line = p.stdout.read()
    else:
        p.poll()
        if p.returncode is not None:
            break

    print('Looping')
    time.sleep(1)

p.wait()

you should use `p.communicate()`, not use stdout directly. See https://docs.python.org/2/library/subprocess.html#subprocess.Popen.stdin ... — Corley Brigman, Jul 06 '17 at 16:48
How are you executing these scripts? Reading from `stdout` is not the usual attempt at inter-process communication. — Prune, Jul 06 '17 at 16:48
@CorleyBrigman Using communicate() is useless with a long-running process. You won't be able to print it until the end. — Dustin Oprea, Jul 06 '17 at 16:49
@Prune IPC is not a requirement. All I need to do is print the output iteratively while capturing it. — Dustin Oprea, Jul 06 '17 at 16:50
This is missing a `sys.stdout.flush()` (or `flush=True`) in the generator. Otherwise, you'll have to wait until it fills the output buffer, which --given the `sleep(1)`-- might take a while. — dhke, Jul 06 '17 at 16:51
@dhke It doesn't matter. I've updated the question to reflect that. — Dustin Oprea, Jul 06 '17 at 17:00
@DustinOprea Repeat: "This is missing a sys.stdout.flush() (or flush=True) **in the generator**". Flushing in the consumer won't help you much, when the buffering happens in the generator script. Try `print(random.random(), flush=True)` — dhke, Jul 06 '17 at 17:02
you are right... maybe take a look at this question? https://stackoverflow.com/questions/19880190/interactive-input-output-using-python — Corley Brigman, Jul 06 '17 at 17:06
@DustinOprea Then either (given you call `print()` as a function) `from __future__ import print_function` at the very top or use `sys.stdout.flush()` after the `print(random.random())`. — dhke, Jul 06 '17 at 17:07

score 0 · Accepted Answer · answered Jul 06 '17 at 17:20

As @dhke mentioned, one of the issues is implicit output-buffering in the producer. If you have the ability to change the producer, and you're willing to, and the production is done by calls to the print-function then just add "flush=True" as an argument to the print function. You can also fall-back to doing a sys.stdout.flush() at key points in the producer.

The second problem appears to be iterating over sys.stdout. This never seems to work for a long-running process. The second and third methods

score 0 · Answer 2 · answered Jul 26 '17 at 10:18

I'm dealing with a similar problem. This is the workaround im currently using to prevent buffering.

proc = subprocess.Popen(['stdbuf', '-o0'] + cmd, stdout=subprocess.PIPE)

The disadvantage of this methood is that it relys on an external Linux command to solve the problem. Have a look in the comments here for a different and native python approach to get rid of the PIPE buffering. Many thanks to @9000 for suggesting both solutions to me.

Python subprocess stdout iterator always blocks

2 Answers2