Can I pass queue object in multiprocessing pool starmap method

Question

There are many questions in SO regarding passing multiple arguments in python multiprocessing Pool's starmap method. But what I want to ask is if I can send a queue object in the method which can be shared between different processes?

I am able to do this using threading and multiprocessing Process method, but not using Pool's starmap method.

from multiprocessing import Process, Queue, Pool


def get_data(pageNo,q):
    q.put(pageNo*pageNo)


if __name__ == "__main__":
    q = Queue()
    p = {}
    no_pages = 5
    pool_tuple = [(x,q) for x in range(1,no_pages)]
    with Pool(processes=3) as pool:
        pool.starmap(get_data, pool_tuple)

What I want is that I should be able to put the result in the queue from different processes. The error I am getting is: Queue objects should only be shared between processes through inheritance

Am I doing something wrong in the way I am passing the queue object? Since if Process supports this, so should the Pool method be able to.

Answer 1

try using Manager() like this:

from multiprocessing import Manager, Pool


def get_data(pageNo, q):
    q.put(pageNo * pageNo)


if __name__ == "__main__":
    m = Manager()
    q = m.Queue()
    p = {}
    no_pages = 5
    pool_tuple = [(x, q) for x in range(1, no_pages)]
    with Pool(processes=3) as pool:
        pool.starmap(get_data, pool_tuple)
    for i in range(1, no_pages):
        print("result", i, ":", q.get())

Output:

result 1 : 1
result 2 : 4
result 3 : 9
result 4 : 16