2

Is there an easy way of gathering the output of a subprocess without actually waiting for it?

I can think of creating a subprocess.Popen() with capturing its stdout, then call p.communicate(), but that would block until the subprocess terminates.

I can think of using subprocess.check_output() or similar, but that also would block.

I need something which I can start, then do other stuff, then check the subprocess for being terminated, and in case it is, takes its output.

I can think of two rather complicated ways to achieve this:

  1. Redirect the output into a file, then after termination I can read the output from that file.
  2. Implement and start a handler thread(!) which constantly tries to read data from the stdout of the subprocess and adds it to a buffer.

The first one needs temporary files and disk I/O which I do not really like in my case. The second one means implementing quite a bit.

I guess there might be a simpler way I couldn't think of yet, or a ready-to-be-used solution in some library I didn't find yet.

Alfe
  • 56,346
  • 20
  • 107
  • 159

2 Answers2

1

What's wrong with calling check_output in a thread?

import threading,subprocess

output = ""

def f():
    global output
    output = subprocess.check_output("ls")  # ["cmd","/c","dir"] for windows


t = threading.Thread(target=f)
t.start()
print('Started')
t.join()
print(output)

note that one could be tempted to use p = subprocess.Popen(cmd,stdout=subprocess.PIPE), wait for p.poll() to be != None and try to read p.stdout afterwards: that only works when the output is small, else you get a deadlock because stdout buffer is full and you have to read it from time to time.

Using p.stdout.readline() would work but would also block if the process doesn't print on a regular basis. If your application prints to the output all the time, then you can consider it as non-blocking and the solution is acceptable.

Jean-François Fabre
  • 137,073
  • 23
  • 153
  • 219
  • I need to start a bunch of processes in parallel. I don't know in which order they are going to finish, but I know that they all are producing output. Whenever one is finishing I want to collect its output and use it at once. With your thread approach I would need to wait for (any out of) a bunch of threads. I think I would do that using a Queue. But that sounds rather complex to me. And to have subprocesses and for each of them an additional thread in the main process sounds like doubling the structures. I think then I'd rather go for the temp file approach. – Alfe Jul 03 '17 at 11:55
  • related: https://stackoverflow.com/questions/375427/non-blocking-read-on-a-subprocess-pipe-in-python?rq=1 – Jean-François Fabre Jul 03 '17 at 12:04
0

I think what you want is an unbuffered stdout stream. With that you will be able to capture the output of your process without waiting for it to finish. You can achieve that with the subprocess.Popen() function and the parameter stdout=subprocess.PIPE.

Try something like this

proc = subprocess.Popen(cmd, shell=True, stdout=subprocess.PIPE)

line = proc.stdout.readline()
while line:
    print line
    line = proc.stdout.readline()
Praind
  • 1,551
  • 1
  • 12
  • 25
  • I need to have several of these in parallel, and I do not know which will be how fast and which will finish when. But I want to use the subprocess's output as soon as the subprocess terminates. – Alfe Jul 03 '17 at 11:56