Python2: multiprocessing using a Pool of workers with multiple arguments

Question

I want to try parallel computing in python-2.x using multiprocessing.Pool.

I came up with the following simple code. Unfortunately, I was not able to produce any error message.

Can someone point me into the right direction of what might be wrong with my code?

import numpy as np
import multiprocessing as mp
import timeit

def fun(i,j):
    return i+j

num=2

num_cores = mp.cpu_count()
p = mp.Pool(num_cores)

results = np.array([])
tasks = np.array([(i, j) for i in range(0,num) for j in range(0, num)])

if __name__ == '__main__':
    results = np.array(p.map(fun,tasks))
    print results

Is the fact, that the `return` in your `fun` function in not indented intentional? Or did that happend while formatting your question for StackOverflow? — Leon, Aug 24 '18 at 13:20
What is the expected output of your program @L.Chau? And for anyone interested in what the actual error is: `TypeError: fun() takes exactly 2 arguments (1 given)` — tafaust, Aug 24 '18 at 19:38

tafaust · Answer 1 · 2018-08-24T19:53:13.657

The main problem you have here is that the tuples you want to pass as arguments to the workers are wrapped in a numpy.ndarray and thus it is just one argument as the numpy.ndarray does not get unfolded.

This should work for you:

from __future__ import print_function
import numpy as np
import multiprocessing as mp
import timeit

def fun(ij):
    s = sum(ij)
    print('{0} + {1} = {2}'.format(ij[0], ij[1], s))
    return s

num=2

num_cores = mp.cpu_count()
p = mp.Pool(num_cores)

results = np.array([])
tasks = np.array([(i, j) for i in range(0,num) for j in range(0, num)])

if __name__ == '__main__':
    results = np.array(p.map(fun,tasks))
    print(results)

The output of this script is:

0 + 0 = 0
0 + 1 = 1
1 + 0 = 1
1 + 1 = 2
[0 1 1 2]

I added the from __future__ import print_function statement, see here what it is good for. And here is the according PEP 3105 -- Make print a function.

Final remarks: A more throughout solution can be found here from user senderle.

Python2: multiprocessing using a Pool of workers with multiple arguments

1 Answers1