Deadlocks while using Queues and multiprocessing

Question

I don't understand this part of the multiprocessing doc (python.org) and I quote:

"An example which will deadlock is the following:

from multiprocessing import Process, Queue

def f(q):
    q.put('X' * 1000000)

if __name__ == '__main__':
    queue = Queue()
    p = Process(target=f, args=(queue,))
    p.start()
    p.join()                    # this deadlocks
    obj = queue.get()

" First, why does it block ? And more surprising, it works perfectly when I try with some smaller values than 1000000 in the definition of f (it works with 10,100,1000,10000, but not with 100000).

Thanks a lot for your help !

score 5 · Accepted Answer · answered Jul 19 '15 at 22:42

5

This example illustrates the behaviour described in 17.2.2.2.

if a child process has put items on a queue (and it has not used JoinableQueue.cancel_join_thread), then that process will not terminate until all buffered items have been flushed to the pipe. This means that if you try joining that process you may get a deadlock unless you are sure that all items which have been put on the queue have been consumed.

answered Jul 19 '15 at 22:42

Pete Kirkham

48,893
5
92
171

Thanks, I understood with both your explanation and the duplicate post. What I don't get is why there is a buffer which gets flushed, and what sets the amount of data after which the data is buffered. But now I can program in peace :) – Labo Jul 19 '15 at 23:23

Deadlocks while using Queues and multiprocessing

1 Answers1