5

I am using pty to read non blocking the stdout of a process like this:

import os
import pty
import subprocess

master, slave = pty.openpty()

p = subprocess.Popen(cmd, stdout = slave)

stdout = os.fdopen(master)
while True:
    if p.poll() != None:
        break

    print stdout.readline() 

stdout.close()

Everything works fine except that the while-loop occasionally blocks. This is due to the fact that the line print stdout.readline() is waiting for something to be read from stdout. But if the program already terminated, my little script up there will hang forever.

My question is: Is there a way to peek into the stdout object and check if there is data available to be read? If this is not the case it should continue through the while-loop where it will discover that the process actually already terminated and break the loop.

Woltan
  • 13,723
  • 15
  • 78
  • 104
  • Just an idea, dont know if this works, but: maybe you could check if `stdout.seek(1, os.SEEK_CUR)` throws an exception and determine that way if there is available data? – Jacob Jun 21 '11 at 08:01
  • @cularis `stdout.seek(1, os.SEEK_CUR)` throws this exception: `IOError: [Errno 29] Illegal seek` – Woltan Jun 21 '11 at 08:33
  • General comment: use `p.poll() is not None` (object identity check) rather than `p.poll() != None` (an equality check, less precise and slower). – Chris Morgan Jun 27 '11 at 08:53
  • While your loop can occasionally block, it should never hang indefinitely, because a file being closed is considered ready-to-read by poll. – Jan Hudec Jun 27 '11 at 08:59
  • 1
    `type(p) is subprocess.Popen` doc for `subprocess.Popen.poll()` is "Check if child process has terminated. Returns returncode attribute." – Dan D. Jun 27 '11 at 09:10
  • @Chris Thx for the comment, for the future I'll make use of the object identify check if it is appropriate like in the case above. – Woltan Jun 27 '11 at 09:15
  • @Jan Unfortunately I have the case, where it blocks. If I am in the line `print stdout.readline()` my script will hang, until some output is written to `stdout`. If the process terminates in the meantime, my script will hang nonetheless. – Woltan Jun 27 '11 at 09:17

2 Answers2

11

Yes, use the select module's poll:

import select
q = select.poll()
q.register(stdout,select.POLLIN)

and in the while use:

l = q.poll(0)
if not l:
    pass # no input
else:
    pass # there is some input
Dan D.
  • 73,243
  • 15
  • 104
  • 123
  • I don't think it does anything different than the poll method of the process handle, i.e. does not solve the problem. – Jan Hudec Jun 27 '11 at 09:04
  • while the poll method of the process handle checks for the end of the process, this would check if stdout has input waiting to be read. – Dan D. Jun 27 '11 at 09:07
  • Ah, ok. If the poll of process checks only that the program exited (hm, how could I think it does anything else -- it does not *know* about the pipe for $deity's sake), it could have never worked reasonably. – Jan Hudec Jun 27 '11 at 13:45
2

The select.poll() answer is very neat, but doesn't work on Windows. The following solution is an alternative. It doesn't allow you to peek stdout, but provides a non-blocking alternative to readline() and is based on this answer:

from subprocess import Popen, PIPE
from threading import Thread
def process_output(myprocess): #output-consuming thread
    nextline = None
    buf = ''
    while True:
        #--- extract line using read(1)
        out = myprocess.stdout.read(1)
        if out == '' and myprocess.poll() != None: break
        if out != '':
            buf += out
            if out == '\n':
                nextline = buf
                buf = ''
        if not nextline: continue
        line = nextline
        nextline = None

        #--- do whatever you want with line here
        print 'Line is:', line
    myprocess.stdout.close()

myprocess = Popen('myprogram.exe', stdout=PIPE) #output-producing process
p1 = Thread(target=process_output, args=(myprocess,)) #output-consuming thread
p1.daemon = True
p1.start()

#--- do whatever here and then kill process and thread if needed
if myprocess.poll() == None: #kill process; will automatically stop thread
    myprocess.kill()
    myprocess.wait()
if p1 and p1.is_alive(): #wait for thread to finish
    p1.join()

Other solutions for non-blocking read have been proposed here, but did not work for me:

  1. Solutions that require readline (including the Queue based ones) always block. It is difficult (impossible?) to kill the thread that executes readline. It only gets killed when the process that created it finishes, but not when the output-producing process is killed.
  2. Mixing low-level fcntl with high-level readline calls may not work properly as anonnn has pointed out.
  3. Using select.poll() is neat, but doesn't work on Windows according to python docs.
  4. Using third-party libraries seems overkill for this task and adds additional dependencies.
Vikram Pudi
  • 1,157
  • 10
  • 6