How to use a Pool of Workers with a Queue in python?

Question

I am currently trying to set up a Parameter Study with multiprocessing, but I don't understand exactly how to use a queue to get the results back to the main process.

Can somebody please help and tell me, why my example doesn't work?

    import numpy as np
    import multiprocessing as mp
    from queue import Empty


    def run(q1):

        q1.put({'val':np.random.random(), 'ErrFlag':False})


    if __name__ == '__main__':

        q = mp.Queue()

        pool = mp.Pool(processes = 10)

        results = []

        for i in range(50):
            pool.apply_async(run, (q,))

        pool.close()
        pool.join()

        while True:
            try:
                res = q.get(timeout = 1)

                results.append(res['val'])

            except Empty:
                break

        print(f'result: {repr(np.array(results))}\n')

When executing this, I only get an empty array.

Answer 1

This is a duplicate of Sharing a result queue among several processes

You should use a manager to make your queue accessible to the different workers:

pool = mp.Pool(processes=10)
m = mp.Manager()
q = m.Queue()