5

There are already a bunch of answers that solve how to do this general thing, but my main question is Why doesnt this approach work?

I am trying to "live stream" the stdout and stderr from a subprocess. I can do this by doing:

import sys
import subprocess

def run_command(cmd):
    process = subprocess.Popen(cmd, stderr=subprocess.PIPE, stdout=subprocess.PIPE)

    for out in iter(process.stdout.readline, b''):
        print(out)

    for err in iter(process.stderr.readline, b''):
        print(err)

run_command(['echo', 'hello', 'world']) # should print hello world
run_command(['rm', 'blarg223'])  # Should print to stderr (file doesnt exist)

and this works giving me the result:

b'hello world\n'
b'rm: cannot remove \xe2\x80\x98blarg223\xe2\x80\x99: No such file or directory\n'

However this causes a problem as it really only live streams the stdout, then prints all the errors as the end. So I thought I could fix this using:

def run_command(cmd):
    process = subprocess.Popen(cmd, stderr=subprocess.PIPE, stdout=subprocess.PIPE)

    for out, err in zip(iter(process.stdout.readline, b''), iter(process.stderr.readline, b'')):
        print(out)
        print(b'Error: ' + err)

But, this doesnt produce any output. Why does using zip not work?

Nick Humrich
  • 14,905
  • 8
  • 62
  • 85
  • Do you expect to always have a one-to-one correspondence between lines of stdout and stderr? You're trying to take lines in pairs. – user2357112 Jan 13 '16 at 21:53
  • both code example may cause deadlock if stdout/stderr do not produce the same amount of output (greater than a threshold) – jfs Jan 15 '16 at 07:00
  • @J.F.Sebastian how so? I have tested it and it seems to work. Can you give an example of how a deadlock would happen? – Nick Humrich Jan 20 '16 at 05:06
  • @NickHumrich: ask a separate question if you can't create an example that produces the deadlock. – jfs Jan 20 '16 at 06:56

1 Answers1

3

zip stops when one of the iterators is finished.

In each of the examples you gave, one stream(stdout/stderr) is empty. So zip will produce nothing.

To fix this you should use itertools.zip_longest

Yoav Glazner
  • 7,936
  • 1
  • 19
  • 36
  • @NickHumrich: `zip_longest()` won't fix the deadlock in the general case. The code in the question is broken. If you want to read stdout/stderr as "live stream" separately; you should read the pipes *concurrently* e.g., [use threads](http://stackoverflow.com/a/31867499/4279), or use [a `select()`-loop](http://stackoverflow.com/a/31953436/4279) – jfs Jan 15 '16 at 07:15