Is there any way to distinguish between not done and blocked with Popen.poll()

Question

I would like to "sow" a number of processes and then "harvest" them when they're done. Using the subprocess module I have the following code in subrun.py:

import time, subprocess, shlex, os

ok = subprocess.Popen(shlex.split("python ok.py"),
                      stdout=subprocess.PIPE,
                      stderr=open(os.devnull, 'w'))


nok = subprocess.Popen(shlex.split("python nok.py"),
                       stdout=subprocess.PIPE,
                       stderr=open(os.devnull, 'w'))

procs = {'ok': ok, 'nok': nok}

while procs:
    running = procs.keys()
    print "running:", running
    for k in running:
        proc = procs[k]
        rc = proc.poll()
        if rc is None:
            pass  # still running
        else:
            del procs[k]
            print proc.stdout.read()

    time.sleep(.4)

ok.py is as follows

import sys
print "OK"
sys.exit(0)

and nok.py is

import sys
print "NOK" * 5000
sys.exit(0)

The output is

(dev) C:\work\dev\test>python subrun.py
running: ['ok', 'nok']
running: ['ok', 'nok']
OK

running: ['nok']
running: ['nok']
running: ['nok']
running: ['nok']
running: ['nok']
running: ['nok']
running: ['nok']
Traceback (most recent call last):
  File "subrun.py", line 27, in <module>
time.sleep(.4)

ie. Popen.poll() returns None when the subprocess has blocked on IO.

I could probably start a thread per process, which would call .communicate()[0], but that seems like a lot of extra book keeping...

Is there any way to get this to work?

jfs · Accepted Answer · 2014-03-19T17:08:49.197

1

If you set stdout=PIPE then you should read the pipe otherwise your child process may block forever if it generates enough output.

It is a bug in your code. Fix it.

To get all output after subprocesses finish:

#!/usr/bin/env python
import shlex
from multiprocessing.dummy import Pool
from subprocess import check_output

cmds = map(shlex.split, ["python ok.py", "python nok.py"])
outputs = Pool(len(cmds)).map(check_output, cmds)

edited Mar 19 '14 at 17:08

answered Mar 19 '14 at 06:24

jfs

399,953
195
994
1,670

How would I go about reading the pipe without blocking if there is nothing to read? (the `subprocess` documentation is a little light on the subject..) – thebjorn Mar 19 '14 at 15:53
[Non-blocking read on a subprocess.PIPE in python](http://stackoverflow.com/q/375427/4279) – jfs Mar 19 '14 at 15:56
I'm guessing you're referring to your answer(?), but isn't that just starting a thread for each process and then having each thread block on read? i.e. it isn't _really_ a non-blocking read. Is it impossible to do a truly non-blocking check of pending output from the subprocess? – thebjorn Mar 19 '14 at 16:09
if you don't like `q.get_nowait()` (that doesn't block (really)) then read the rest of the answers that use select, fcntl, iocp, etc – jfs Mar 19 '14 at 16:17
I'm not really interested in the process until it's finished, and then I would like to get _all_ of its output at once, so it just seems like a lot of book-keeping for something that's conceptually much simpler... ;-) – thebjorn Mar 19 '14 at 16:57
use `check_output()` to get all output after the process finishes. I've added code example – jfs Mar 19 '14 at 17:04

Is there any way to distinguish between not done and blocked with Popen.poll()

1 Answers1