0

I am working in a Jupyter notebook. I'm new to multiprocessing in python, and I'm trying to parallelize the calculation of a function for a grid of parameters. Here is a snippet of code quite representative of what I'm doing:

import os
import numpy as np
from concurrent.futures import ProcessPoolExecutor

def f(x,y):
    print(os.getpid(), x,y,x+y)
    return x+y

xs = np.linspace(5,7,3).astype(int)
ys = np.linspace(1,3,3).astype(int)

func = lambda p: f(*p)
with ProcessPoolExecutor() as executor:
    args = (arg for arg in zip(xs,ys))
    results = executor.map(func, args)
    
for res in results:
    print(res)

The executor doesn't even start.

No problem whatsoever if I serially execute the same with, e.g. list comprehension,

args = (arg for arg in zip(xs,ys))
results = [func(arg) for arg in args]
berberto
  • 91
  • 1
  • 8

1 Answers1

2

Are you running on Windows? I think your main problem is that each process is trying to re-execute your whole script, so you should include an if name == "main" check. I think you have a second issue trying to use a lambda function that can't be pickled, since the processes communicate by pickling the data. There are work-arounds for that but in this case it looks like you don't really need the lambda. Try something like this:

import os
import numpy as np
from concurrent.futures import ProcessPoolExecutor


def f(x, y):
    print(os.getpid(), x, y, x + y)
    return x + y

if __name__ == '__main__':

    xs = np.linspace(5, 7, 3).astype(int)
    ys = np.linspace(1, 3, 3).astype(int)

    with ProcessPoolExecutor() as executor:
        results = executor.map(f, xs, ys)

    for res in results:
        print(res)
brensnap
  • 129
  • 2
  • I am on Linux and it's running in jupyter. The `__main__` thing is not an issue. The code with the map to two lists actually works. Thanks! – berberto Nov 25 '20 at 23:06
  • Pickle just can't handle a lambda (or a non-top level function). Can you define a function in the top level of your module that calls the lambda, or change the lambda to a regular function? – brensnap Nov 25 '20 at 23:20
  • If not, check out all of the answers on this post. I am not sure which of the libraries they suggest have been updated recently. https://stackoverflow.com/questions/8804830/python-multiprocessing-picklingerror-cant-pickle-type-function?noredirect=1&lq=1 – brensnap Nov 25 '20 at 23:26
  • Yes, definitely, I've just changed `func` into a regular function, and that works fine. I had removed my question from my previous comments, realizing it was stupid :) Thanks – berberto Nov 26 '20 at 00:12