Use multiprocessing (multiple cores or multiple threads) to calculate my function

Question

I use the code below to use multiple cores of CPUs to calculate a function (it is an image processing function). However, when the process is finished, the returned value is an object. I expect to see a NumPy array. Any suggestion is appreciated.

import datetime
from multiprocessing import Process, current_process
import sys
...

def my_function(ImgDir, MaskDir, False, True):
    ...
    All_numpy_arrays = np.dstack(foregrounds)
    return All_numpy_arrays

if __name__ == '__main__':
    worker_count = 6
    worker_pool = []
    p = None
    for _ in range(worker_count):
        result = Process(target=my_function,
                    args=(ImgDir, MaskDir, False, True))
        result.start()
        worker_pool.append(result)
    for result in worker_pool:
        result.join()  

    # Here I have another script that requires the *result* be a NumPy array

If it helps, I use Ubuntu 20.04 and python 3.8.

Any reason you are not using `map()` as demonstrated in the [docs](https://docs.python.org/3/library/multiprocessing.html) (1st example)? — Klaus D., Aug 03 '20 at 16:06
You can't return data directly from a process. For that, you need some sort of shared variable. Take a look at [this](https://stackoverflow.com/questions/10415028/how-can-i-recover-the-return-value-of-a-function-passed-to-multiprocessing-proce) for some methods, such as Managers or Queues. — rcole, Aug 03 '20 at 15:52

score 2 · Answer 1 · answered Aug 03 '20 at 16:08

You could use multiprocessing.Queue like so:

import multiprocessing

def runme(i, q):
    q.put(i)

if __name__ == '__main__':
    multiprocessing.freeze_support() # windows support
    queue = multiprocessing.Queue()
    ps = [multiprocessing.Process(target=runme, args=[i, queue]) for i in range(6)]
    [p.start() for p in ps]
    [p.join() for p in ps]
    while True:
        try:
            print(queue.get_nowait())
        except multiprocessing.queues.Empty:
            break

The above outputs something like:

Use multiprocessing (multiple cores or multiple threads) to calculate my function

1 Answers1