Trouble with blocking while using Join() with Multiprocess

Question

I don't understand why the following code blocks. I'm trying to simply loop for a set amount of time, and then send a message to the thread to stop. My process class is as follows.

class Worker(multiprocessing.Process):
    def __init__(self, queue):
        multiprocessing.Process.__init__(self)
        self.queue = queue
        self.running = True

    def run(self):
        print 'entered run'
        while self.running:
            print 'thread time:', time.time()
            time.sleep(.6)
        print 'thread end'
        return 0

Pretty straight forward. My main looks like this:

if __name__ == '__main__':
    queue = Queue()
    p = Worker(queue)
    p.daemon = True
    p.start()
    time.sleep(3)
    p.running = False
    print 'Main end'

So, what I expect this program to do is to start the process which will run its own loop along side of main's. While that is happening, Main() will sleep for 3 seconds, then set the loop condition inside of the worker process to False so it breaks out of the loop. Then they both print a nice "done" message, and the program exits. However, the problem is that the main thread exits before the process has a chance to show its message.

For example, the output looks like this:

>> entered run
>> thread time: 1358444245.31
>> thread time: 1358444245.91
>> thread time: 1358444246.51
>> thread time: 1358444247.11
>> thread time: 1358444247.71
>> Main end

So, reading a little, and I see that join() is used to block until a thread finishes execution. Following this, I add a join() call right after I break the loop condition in the thread. My thinking is that this will block main until the process has time to close itself down.

if __name__ == '__main__':
    queue = Queue()
    p = Worker(queue)
    p.daemon = True
    p.start()
    time.sleep(3)
    p.running = False
    p.join()          ## <------ New Line 
    print 'Main end'

However, this behaves unexpectedly. Once I add that line, the thread never stops executing, and so main blocks indefinitely. The below output will spin out into eternity.

>> entered run
>> thread time: 1358444362.44
>> thread time: 1358444363.04
>> thread time: 1358444363.64
>> thread time: 1358444364.24
>> thread time: 1358444364.84
>> thread time: 1358444365.44
>> thread time: 1358444366.04
>> thread time: 1358444366.64
>> thread time: 1358444367.24
>> thread time: 1358444367.84
>> thread time: 1358444368.44
>> thread time: 1358444369.04
>> thread time: 1358444369.64
>> thread time: 1358444370.24

What's going on here? Why when I add join() does my thread not break out of its loop?

See http://stackoverflow.com/questions/6359597/gracefully-terminating-python-threads for interthread communication via events. In your case I guess (as in other languages) you are probably seeing a cached value of the flag used. — Max Yakimets, Jan 17 '13 at 17:55

score 2 · Answer 1 · answered Jan 17 '13 at 18:30

First, multiprocessing.Process does not create a Thread, it creates a separate process that will execute a provided function.

You will need to use multiprocessing.Queue to pass messages between the 2 processes.

Take a look at: http://docs.python.org/2/library/multiprocessing.html#exchanging-objects-between-processes

As Max Yakimets mentionned, you are modifying values within the first process but never passing a message to the second one.

Instead of changing the p.running value, you will need to add an item on the queue. Within the process, you will need to check if there is any messages waiting within the queue and react to them (shutting down, doing somethign else, etc.)

Trouble with blocking while using Join() with Multiprocess

1 Answers1