Python: How to read stdout of subprocess in a nonblocking way

Question

I am trying to make a simple python script that starts a subprocess and monitors its standard output. Here is a snippet from the code:

process = subprocess.Popen([path_to_exe, os.path.join(temp_dir,temp_file)], stdout=subprocess.PIPE)
while True:   
    output=process.stdout.readline()
    print "test"

The problem is that the script hangs on output=process.stdout.readline() and that the line print "test" only executes after the subprocess is terminated.

Is there a way to read standard output and print it without having to wait for the subprocess to terminate?

The subprocess which I am starting is a Windows binary for which I do not have the source code.

I have found several similar questions, but the answers are only applicable on Linux or in case I have the source of the suprocess I am starting.

I think you need to provide more details about your executable. What is the sample output of the binary? A single line, or many? It kind of sounds like the binary is poorly suited for such an interface. In that the output is only flushed as a consequence of the binary terminating. — Dunes, Apr 07 '16 at 13:51
I actually want to fuzz different binaries (Acrobat reader but also other) and to detect crashes. I can get the Exit code, but I would also like to have the standard output/error. — pineappleman, Apr 07 '16 at 13:57

score 13 · Accepted Answer · edited Jun 20 '20 at 09:12

13

Check select module

import subprocess
import select
import time
    
x=subprocess.Popen(['/bin/bash','-c',"while true; do sleep 5; echo yes; done"],stdout=subprocess.PIPE)
    
y=select.poll()
y.register(x.stdout,select.POLLIN)

while True:
  if y.poll(1):
     print x.stdout.readline()
  else:
     print "nothing here"
     time.sleep(1)

EDIT:

Threaded Solution for non posix systems:

import subprocess
from threading import Thread 
import time
 
linebuffer=[]
x=subprocess.Popen(['/bin/bash','-c',"while true; do sleep 5; echo yes; done"],stdout=subprocess.PIPE)

def reader(f,buffer):
   while True:
     line=f.readline()
     if line:
        buffer.append(line)
     else:
        break

t=Thread(target=reader,args=(x.stdout,linebuffer))
t.daemon=True
t.start()

while True:
  if linebuffer:
     print linebuffer.pop(0)
  else:
     print "nothing here"
     time.sleep(1)

edited Jun 20 '20 at 09:12

Community

1
1

answered Apr 07 '16 at 13:21

xvan

4,554
1
22
37

I get the message "AttributeError: 'module' object has no attribute 'poll'. Could it be that that module is different for Windows? – pineappleman Apr 07 '16 at 13:48
1

Seems like select can't work with streams on windows, [here](http://stackoverflow.com/a/22254123/1477064) threads are suggested as the next best option. – xvan Apr 07 '16 at 13:52
Seems like you'll be printing output lines in reverse, if you get several lines in one poll. using `pop(0)` instead of `pop()` would fix it – Max Yankov Jan 16 '19 at 14:10
I found it surprising that when I removed `print linebuffer.pop(0)`, the program still prints subprocess outputs, where is the output from? – desmond ng Jul 23 '21 at 15:51
maybe it's printing to stderr? – xvan Jul 23 '21 at 16:08
Instead of while(true) I would propose while x.poll() so when the program terminates, the loop finishes – MappaM Jun 14 '23 at 15:38

score 1 · Answer 2 · edited Nov 28 '22 at 16:11

As of python 3.5, you can use os.set_blocking() as follows.

import os
import subprocess

process = subprocess.Popen([path_to_exe, os.path.join(temp_dir,temp_file)], stdout=subprocess.PIPE)
os.set_blocking(process.stdout.fileno(), False) #<----- HERE
while True:   
    output=process.stdout.readline()
    print "test"


    if process.poll() is not None:
        break
rc = process.poll()

Note that os.set_blocking() is only available on UNIX platforms.

score -1 · Answer 3 · answered Apr 07 '16 at 13:10

You could try this:

import subprocess
import os

""" Continuously print command output """
""" Will only work if there are newline characters in the output. """

def run_cmd(command):    
    popen = subprocess.Popen(command, stdout=subprocess.PIPE)
    return iter(popen.stdout.readline, b"")

for line in run_cmd([path_to_exe, os.path.join(temp_dir,temp_file)]):
    print(line), # the comma keeps python from adding an empty line

Python: How to read stdout of subprocess in a nonblocking way

3 Answers3

EDIT:

Linked