0

I would like to run different functions in parallel (foo, bar, and baz), and then get the value returned by each function in the order in which they were called.

I tried something like the code below, but fore some reason it loops forever on Windows. Is it possible to achieve this using for example multiprocessing ? Note that the functions are expected to take as parameter the same thing (i.e. data).

Note: The code below is for Python 3, I want an equivalent for Python 2.7 as it is what I need to use.

from concurrent.futures import ProcessPoolExecutor
from operator import itemgetter

def foo(data):
    return "foo"

def bar(data):
    return "bar"

def baz(data):
    return "baz"

work = [foo, bar, baz]
data = [1,2,3,4,5]

results = []

with ProcessPoolExecutor(max_workers=4) as pool:
    for i, work_item in enumerate(work):
        future = pool.submit(work_item, data)
        def callback(ret):
            results.append((i, ret.result()))
        future.add_done_callback(callback)

results.sort(key=itemgetter(0))
print(results)
eLearner
  • 325
  • 6
  • 18

1 Answers1

1

I changed your callback method a bit and achieved what you want, I guess. Check the result:

...
with ProcessPoolExecutor(max_workers=4) as pool:
    for i, work_item in enumerate(work):
        future = pool.submit(work_item, data)
        def callback(i):
            def wrapper(ret):
                print(ret, i)
                results.append((i, ret.result()))
            return wrapper
        future.add_done_callback(callback(i))
...

and the result is as follows:

<Future at 0x1057e4d30 state=finished returned str> 0
<Future at 0x1057e4a58 state=finished returned str> 2
<Future at 0x105779240 state=finished returned str> 1
[(0, 'foo'), (1, 'bar'), (2, 'baz')]

I run this code on a mac os x with python3. I hope it helps.

for python 2.7, you can try something like this:

from multiprocessing import Process, Manager

def foo(data, i, results):
    results[i] = "foo"

def bar(data, i, results):
    results[i] = "bar"

def baz(data, i, results):
    results[i] = "baz"

work = [foo, bar, baz]
data = [1,2,3,4,5]

processes = []
results = Manager().dict()
for i, w in enumerate(work):
    p = Process(target=w, args=(data, i, results))
    processes.append(p)
    p.start()

for p in processes:
    p.join()

print results

result is:

{0: 'foo', 1: 'bar', 2: 'baz'}

How can I recover the return value of a function passed to multiprocessing.Process?

Community
  • 1
  • 1
Muatik
  • 4,011
  • 10
  • 39
  • 72
  • I forgot to mention that the given code is for Python 3 (and doesn't work on Windows), but I want a code which works with Python 2.7 as it is what I need to use. – eLearner Apr 11 '17 at 19:40
  • then have a look at this page: https://pymotw.com/2/multiprocessing/communication.html – Muatik Apr 11 '17 at 20:16
  • Looks good, thanks. I think that it would be also interesting to limit the number of created process (running in parallel) as it will slow down if this number is much higher than the number of cores. Or is that automatically managed by `multiprocessing`? – eLearner Apr 19 '17 at 09:47