Non-blocking read on a subprocess.PIPE in Python

Question

I read the question/answer/comments on A non-blocking read on a subprocess.PIPE in Python, but I felt a bit lacking.

When I implemented the solution provided, I noticed that this approach works best when the sub-process ends on it own. But if the subprocess is providing a stream of information and we are looking for a single match of output, then that approach doesn't work for my needs (specifically for Windows, if that matters).

Here is my sample:

File ping.py

import time

def main():
    for x in range(100):
        print x
        time.sleep(1)

if __name__ == '__main__':
    print("Starting")
    time.sleep(2)
    main()

File runner.py

import subprocess
import time
import sys
from Queue import Queue, Empty
from threading  import Thread

def enqueue_output(out, queue):
    for line in iter(out.readline, b''):
        queue.put(line)
    out.close()

# Start process we want to listen to
pPing = subprocess.Popen('ping.py',
    shell=True,
    stdin=subprocess.PIPE,
    stdout=subprocess.PIPE,
    )

q = Queue()
t = Thread(target=enqueue_output, args=(pPing.stdout, q))
t.daemon = True
t.start()

# Make sure it's started
print ("get the first line")
try:
    line = q.get()
except Empty:
    pass
else:
    print line.strip()

#look for the 'magic' output
print("empty the queue")
while not q.empty():
    line = q.get_nowait().strip()
    if (line == "3"):
        print("got it!")
        sys.exit()
    else:
        print("not yet")

My expectation is that the runner will make sure the process is started and then wait for the magic output and then stop, which it does. However, the longer the sub-process runs, the longer the runner runs. But since the 'magic' output comes relatively quickly, I have to wait until the subprocess ends before I get anything processed.

What am I missing?

Answer 1

OK, if I understand correctly what you are trying to do, the problem is with ping still being a child process to runner. While you can make read calls non-blocking, the parent process will not actually exit while the child is still running. If you want runner not to wait for the child to finish, read the first line and the first magic output and then exit; you need ping to disassociate itself from the parent process.

Look at this code sample to see how that is done A simple Unix/Linux daemon in Python. Of course you might skip the part where they close and re-open all the I/O streams.

On the same note, I am not sure leaving an open I/O stream connected to the parent will allow the parent to exit, so if that happens to be a problem you might have to figure out another way to exchange data.