MultiProcessing in python3 not working

Question

I am using multi processing in python3 for first time. Here is what I am trying to implement :

import concurrent
t = concurrent.futures.ProcessPoolExecutor(4)
g = t.map(lambda x:10*x, range(10))

And it is throwing errors:

File "/usr/local/Cellar/python3/3.4.3/Frameworks/Python.framework/Versions/3.4/lib/python3.4/multiprocessing/queues.py", line 242, in _feed
    obj = ForkingPickler.dumps(obj)
  File "/usr/local/Cellar/python3/3.4.3/Frameworks/Python.framework/Versions/3.4/lib/python3.4/multiprocessing/reduction.py", line 50, in dumps
    cls(buf, protocol).dump(obj)
_pickle.PicklingError: Can't pickle <function <lambda> at 0x102f03730>: attribute lookup <lambda> on __main__ failed

and hangs out every time.

score 3 · Accepted Answer · edited May 23 '17 at 12:23

The function passed to t.map needs to be picklable. Anonymous functions (i.e. lambda functions) are not picklable -- at least not by default. To fix, define the function in the global namespace:

import concurrent.futures as CF
def func(x):
    return 10*x

if __name__ == '__main__':
    with CF.ProcessPoolExecutor(4) as t:
        g = t.map(func, range(10))
        print(list(g))
        # [0, 10, 20, 30, 40, 50, 60, 70, 80, 90]

MultiProcessing in python3 not working

1 Answers1