Python multiprocessing windows and jupyter

Question

I am trying to run some multiprocessing routine using pathos package. Here is my code:

import multiprocessing as mp
import time

def square(x):
    return x*x

if __name__=="__main__":
    start_time = time.time()
    p = mp.Pool(3)
    function_results = p.map(square,[1,2,3]) #ordered
    p.close()
    print("--- %s seconds ---" % (time.time() - start_time))

I am trying it to run it on Win 10 and using Jupyter notebook.

The code does not run and generates the following error:

Process SpawnPoolWorker-2:
Traceback (most recent call last):
  File "C:\Users\user\AppData\Local\conda\conda\envs\tensorflow\lib\multiprocessing\process.py", line 252, in _bootstrap
    self.run()
  File "C:\Users\user\AppData\Local\conda\conda\envs\tensorflow\lib\multiprocessing\process.py", line 93, in run
    self._target(*self._args, **self._kwargs)
  File "C:\Users\user\AppData\Local\conda\conda\envs\tensorflow\lib\multiprocessing\pool.py", line 108, in worker
    task = get()
  File "C:\Users\user\AppData\Local\conda\conda\envs\tensorflow\lib\multiprocessing\queues.py", line 337, in get
    return ForkingPickler.loads(res)
AttributeError: Can't get attribute 'square' on <module '__main__' (built-in)>

This code ran fine on Ubuntu 16.04. Any suggestions on how to fix it?

@georgexsh, can you try it through Jupyter? I tried on Win 7 and got the same error — user1700890, Dec 14 '17 at 18:12
nope, but I suggest you do not use multiprocessing with jupyter notebook, behind the scene, jupyter have complex internal logic, which would not play well with mp very likely. — georgexsh, Dec 14 '17 at 18:15

score 3 · Accepted Answer · answered Jan 03 '18 at 16:46

3

Try put the function square in a separate file then import it, as mentioned in this post

answered Jan 03 '18 at 16:46

cpc

149
4

Python multiprocessing windows and jupyter

1 Answers1