How to share queue between parent and child in Python multipleprocssing

Question

I want to save the task result in to a queue, the tasks is run in multiple process. After all tasks running done, read result from the queue.

Why the queue size printed in child task is right, but in parent process is 0? Also I can not get the queue data in parent. How to resolve this?

Code in Python3:

import multiprocessing

que = multiprocessing.Queue()

def task(n):
    que.put(n)
    print("Child: queue len is %d" % que.qsize())

if __name__ == '__main__':
    pool = multiprocessing.Pool()
    pool.map(task, range(10))
    print("Parent: queue len is %d" % que.qsize())

Output:

Child: queue len is 1
Child: queue len is 2
Child: queue len is 3
Child: queue len is 4
Child: queue len is 5
Child: queue len is 6
Child: queue len is 7
Child: queue len is 8
Child: queue len is 9
Child: queue len is 10
Parent: queue len is 0    // Why here is not 10??

zvone · Accepted Answer · 2018-05-26T15:56:42.020

When you run task in pool.map, the module containing function task is imported and task is called with arguments you specified in pool.map.

So, the original queue is not passed to it. When you import the module (in the other process), a new global queue is created.

To make the behaviour more apparent, I modified you example a bit, so that it spawns more processes:

import multiprocessing
import time

que = multiprocessing.Queue()

def task(n):
    que.put(n)
    print("Child: queue len is %d" % que.qsize())
    time.sleep(0.1)

if __name__ == '__main__':
    pool = multiprocessing.Pool(10)
    pool.map(task, range(10))
    print("Parent: queue len is %d" % que.qsize())

Now it prints:

Child: queue len is 1
Child: queue len is 1
Child: queue len is 2
Child: queue len is 1
Child: queue len is 1
Child: queue len is 1
Child: queue len is 1
Child: queue len is 1
Child: queue len is 1
Child: queue len is 1
Parent: queue len is 0

To share the queue, you would have to pass it to the other processes, but cannot be done through map. There alternatives offered in several other SO answers, e.g.

What is best in your case depends on what you are actually trying to do with that queue.

Thank you, it make sense for me – chenchang Jun 20 '18 at 06:01 — chenchang, Jun 20 '18 at 06:01

How to share queue between parent and child in Python multipleprocssing

1 Answers1