4

Say I write this:

from subprocessing import Popen, STDOUT, PIPE
p = Popen(["myproc"], stderr=STDOUT, stdout=PIPE)

Now if I do

line = p.stdout.readline()

my program waits until the subprocess outputs the next line.

Is there any magic I can do to p.stdout so that I could read the output if it's there, but just continue otherwise? I'm looking for something like Queue.get_nowait()

I know I can just create a thread for reading p.stdout, but let's assume I can't create new threads.

itsadok
  • 28,822
  • 30
  • 126
  • 171

2 Answers2

8

Use p.stdout.read(1) this will read character by character

And here is a full example:

import subprocess
import sys

process = subprocess.Popen(
    cmd, stdout=subprocess.PIPE, stderr=subprocess.PIPE
)

while True:
    out = process.stdout.read(1)
    if out == '' and process.poll() != None:
        break
    if out != '':
        sys.stdout.write(out)
        sys.stdout.flush()
Nadia Alramli
  • 111,714
  • 37
  • 173
  • 152
  • THANK YOU! I don't know how many questions I've seen asking for a simple answer to this issue - this was it. I'd been trying to read character by character too but I didn't know how to check if the process had finished....completely missed poll() - duh. +1 – jkp Jan 15 '11 at 17:03
  • 1
    +1 this solution might be appropriate on Windows where `select` doesn't work on file descriptors created by `subprocess` module. Note: if the subprocess generates enough output on stderr then it might block unless you read from it (OP redirects stderr to stdout (`stderr=STDOUT`) so this won't happen). Use `is` to compare with `None` e.g., `if x is not None:`. You could use `if out:` instead of `if out != '':` (it works both for bytestrings and Unicode strings). Also if `cmd` uses block-buffering in non-interactive mode then `process.stdout` won't get anything until `cmd` flushes the buffer. – jfs Apr 03 '13 at 07:26
  • 4
    I don't think this answers the question that was asked. This method blocks at the read(1) if there's no output available-- what was asked for was a method that does not block in that case, but instead goes on to do something else. – Don Hatch Jul 02 '15 at 13:33
  • @DonHatch you're correct, but it was useful to me anyway – Erik Aronesty Jul 28 '20 at 20:07
5

Use the select module in Python's standard library, see http://docs.python.org/library/select.html . select.select([p.stdout.fileno()], [], [], 0) immediately returns a tuple whose items are three lists: the first one is going to be non-empty if there's something to read on that file descriptor.

Alex Martelli
  • 854,459
  • 170
  • 1,222
  • 1,395
  • What exactly is returned? This still doesn't give a clue how much bytes is there to read. – iElectric Aug 24 '09 at 22:23
  • 2
    `select` returns the set of file descriptors (among those you passed it) on which I/O can be performed without blocking. If you can arrange for p.stdout to be set to non-blocking after the fact (system dependent) you can just p.stdout.read(N) for some large N and get all bytes that are in fact available; if the nonblocking option is not available, all you can do is read 1 byte at a time and go back to `select`ing until that tells you there are no more. Be sure to append the bytes you read into a list and ''.join it at the end (much faster than +='ng away on strings!-). – Alex Martelli Aug 24 '09 at 22:33
  • `select` should not work on windows with file descriptors according to docs in 2.x. – n611x007 Sep 09 '13 at 07:31
  • `from subprocess import Popen, PIPE, STDOUT ; p=Popen(['python.exe'], stdout=PIPE, stderr=STDOUT, stdin=PIPE) ; select((p.stdout.fileno(),), (), (), 0)` produces `error: (10038, 'An operation was attempted on something that is not a socket')` because, as the python documentation says, select.select is implemented using WinSock's select, which only accepts sockets, not file descriptors. – kfsone Dec 21 '17 at 01:42