How to read stream of a subprocess in python?

Question

I create a process with subprocess.Popen and get its stdout. I want to read content from stdout and print it out in another thread. Since my purpose it to make a an interactive program later, I cannot use subprocess.communicate.

My basic requirement is: Once subprocess output something, the thread should immediately print it out to the screen.

Here is the code

import subprocess
import thread
import time

def helper(s):
    print "start",time.gmtime()
    while True:
        print "here"
        print s.next()
    print "finished",time.gmtime()
    thread.start_new_thread(helper,(out_stream,))

def process_main():
    global stdin_main
    global stdout_main
    p = subprocess.Popen("/bin/bash", shell=False, stdin=subprocess.PIPE,
                         stdout=subprocess.PIPE,
                         stderr=subprocess.STDOUT, bufsize=0)
    stdin_main = p.stdin
    stdout_main = p.stdout
    th1 = thread_print_stream(stdout_main)
    stdin_main.write("ls -la\n")
    stdin_main.flush()
    time.sleep(30)

    p.terminate()

process_main()

Time elapse between "start" and "finished" should be very fast. However it is 30 seconds which is exactly the same as time before process terminated. I cannot understand why the output is not instance. Or how can I make it instancely?

You are probably suffering from buffering. This might help: http://stackoverflow.com/questions/1183643/unbuffered-read-from-process-using-subprocess-in-python — cdarke, Sep 29 '16 at 08:04
@cdarke suffering from subprocess or file? I have set bufsize=0. — worldterminator, Sep 29 '16 at 08:13
@cdarke Yes. it has nothing to do with that correct answer, though I did find answer in that thread. I have post an answer to this question. Thx — worldterminator, Sep 29 '16 at 09:47

score 0 · Accepted Answer · answered Sep 29 '16 at 09:45

As cdarke mentioned. This program is suffered from buffering. However it is more like a bug in python 2.x. The logic is nothing wrong in the current program.

To fix the issue, you have to reopen the stdout. Like the code below:

def thread_print_stream(out_stream):
  def helper(s):
    print "start",time.gmtime()
    for line in io.open(out_stream.fileno()):
      print line
    print "finished",time.gmtime()
  thread.start_new_thread(helper,(out_stream,))

And add

stdin_main.close()
stdout_main.close()

before subprocess terminated, to make sure no error rise.

How to read stream of a subprocess in python?

1 Answers1