Python calling pipe.communicate() in a thread

Question

Using Python 2.6.1 on Mac OS X 10.6.2, I've the following problem:

I have a threaded process (a Thread class), and each of those threads has a pipe (a subprocess.Popen) something likeso:

 from threading import Thread

 cmd = "some_cmd"

 class Worker(Thread):
    def run(self):
       pipe = Popen(cmd,
        stdin=PIPE,
        stdout=PIPE,
        stderr=PIPE)

       out, err = pipe.communicate("some data")

The problem is that the pipe.communicate() code is blocking. Interestingly, when I sent an interrupt (e.g. Ctrl-C KeyboardInterrupt) to the parent process then it unblocks.

Interestingly, when I use class Worker(multiprocessing.Process), the code works just fine.

Any thoughts as to why this is blocking - and how to fix it - would be greatly appreciated.

Thank you.

Just found http://stackoverflow.com/questions/984941/python-subprocess-popen-from-a-thread - however I'm not sure from that what the fix is (yet)! — Brian M. Hunt, Mar 30 '10 at 02:08
Looks like Popen might not be thread-safe. Be much obliged for a correction or clarification. Found http://mail.python.org/pipermail/python-bugs-list/2007-August/039177.html on the topic. — Brian M. Hunt, Mar 30 '10 at 02:14
communicate() is *supposed* to block. Do you mean that it blocks the whole program and not just the thread it's running on? Could you provide a complete example program? — Daniel Stutzbach, Mar 30 '10 at 04:37
@Daniel Stutzbach: `communicate` is blocking even though the process does (or ought to) terminate. I'll put up a (non-)working example as soon as I can. — Brian M. Hunt, Mar 30 '10 at 12:09
@Daniel Stutzbach: Because of this problem I gave up and went back to `multiprocessing.Process` (which didn't block on `communicate`) with a quite-different design, so I can't reproduce the test-case from memory. Alas, the cause of this problem will remain a mystery, but I strongly suspect it's because the `Popen` call wasn't thread-safe because the only difference in the design-as-it-was was that it was a `threading.Thread` versus a `multiprocessing.Process`. In the prior the `communicate` blocked (if that's the right word for what was happening), but in the latter it terminated as expected. — Brian M. Hunt, Mar 31 '10 at 01:38

score 1 · Answer 1 · answered Mar 30 '10 at 06:57

1

Using multiple threads and multiple processes will often cause problems, particularly (though not exclusively) on Unix-based systems; I recommend you just avoid mixing the two.

answered Mar 30 '10 at 06:57

Alex Martelli

854,459
170
1,222
1,395

score 1 · Accepted Answer · edited Feb 12 '12 at 13:17

If you call sys.exit() in the main thread, other threads will terminate at the next opportunity (on most operating systems). However, if they're in a blocking call (such as communicate()), they will wait until the blocking call completes before terminating. Control-C works because it causes the operating system to interrupt the blocking call.

Generally, the safest approach is to ensure that none of your threads call functions that may block indefinitely. Unfortunately, that often involves writing a lot more code.

In your particular case, you could store the Popen objects in a global set() before calling communicate, and have the main thread call Popen.terminate() on each of them just before exiting. That would cause the child to exit, communicate() to return, and the thread to exit.

Are you setting the thread as a daemon thread?

Python calling pipe.communicate() in a thread

2 Answers2