I know that there are differences with how python's multiprocessing on Mac and Windows; for example:
these posts seem to imply that the issue is to do with how functions that are not in their own module are pickled, although this answer is a little more specific, and says that the issue is with bound methods.
On Windows, I can start a Jupyter Notebook, and pickle a function I define there:
import pickle
def test(): return None
pickle.dumps(test)
and returns b'\x80\x03c__main__\nt\nq\x00.'
(as expected, this also works directly in a standard interactive python session).
However, wrapping this within a multiprocessor call does, as expected, generate an error (in both a notebook and terminal):
> import multiprocessing as mp
> def test(): return None
> p = mp.Pool(processes=2)
> p.map(test, range(2))
AttributeError: Can't get attribute 'test' on <module '__main__' (built-in)>
This does work in an interactive session on a Mac (although the function defined interactively needs an argument to prevent an error of TypeError: test() takes 0 positional arguments but 1 was given.)
This must mean I haven't got a total grasp on what multiprocessing is doing - I thought it was just pickling the objects, but given I can pickle objects defined in a notebook, multiprocessing must be doing more than this. Can anybody fill in the gaps in my understanding?!
