Why Can't Jupyter Notebooks Handle Multiprocessing on Windows?

Question

On my windows 10 machine (and seemingly other people's as well), Jupyter Notebook can't seem to handle some basic multiprocessing functions like pool.map(). I can't seem to figure out why this might be, even though a solution has been suggested to call the function to be mapped as a script from another file. My question, though is why does this not work? Is there a better way to do this kind of thing beyond saving in another file?

Note that the solution was suggested in a similar question here. But I'm left wondering why this bug occurs, and whether there is another easier fix. To show what goes wrong, I'm including below a very simple version that hangs on my computer where the same function runs with no problems when the built-in function map is used.

import multiprocessing as mp

# create a grid 
iterable = [3, 5, 10]

def add_3(iterable):
    a = iterable + 3
    return a

# Below runs no problem
results = list(map(add_3, iterable))
print(results)

# multiprocessing attempt (hangs)
def main():
    pool = mp.Pool(2)
    results = pool.map(add_3, iterable)    
    return results

if __name__ == "__main__":  #Required not to spawn deviant children    
    results = main()

Edit: I've just tried this in Spyder and I've managed to get it to work. Unsurprisingly running the following didn't work.

if __name__ == "__main__":  #Required not to spawn deviant children    
    results = main()
print(results)

But running it as the following does work because map uses the yield command and isn't evaluated until called which gives the typical problem.

if __name__ == "__main__":  #Required not to spawn deviant children    
    results = main()
    print(results)

edit edit:

From what I've read on the issue, it turns out that the issue is largely because of the ipython shell that jupyter uses. I think there might be an issue setting name. Either way using spyder or a different ide solved the problem, as long as you're not still running the multiprocessing function in an iPython shell.

score 1 · Answer 1 · answered Jun 30 '20 at 04:56

I faced a similar problem like this. I can't use multiprocessing with function on the same script. The solution that works is to put the function on different notebook file and import it use ipynb:

from ipynb.fs.full.script_name import function_name

pool = Pool()
result = pool.map(function_name,iterable_argument)

Why Can't Jupyter Notebooks Handle Multiprocessing on Windows?

1 Answers1