Multiprocessing example giving AttributeError on Mac

Question

Recently I reinstalled the latest Anaconda3 on Mac and found some an error in

from multiprocessing import Pool
def f(x):
    return x*x

with Pool(5) as p:
    p.apply(f, [1])

The error is like

AttributeError: Can't get attribute 'f' on <module '__main__' (built-in)>

But the same code worked several days before!!!

your specific error is from trying to run multiprocessing code in an interactive mode using "spawn" as the startmethod (the new standard method for MacOS). you will run into new errors also without `if __name__ == "__main__":` as well due to the change. Many questions here related to Windows specifically now apply to MacOS as well (windows only has "spawn" whereas Mac and Linux also have "fork". Mac used to use "fork" by default, and linux still does.) — Aaron, Oct 08 '21 at 19:45

score 0 · Answer 1 · answered Oct 08 '21 at 09:08

0

Try this:

from multiprocessing import Pool
def f(x):
    return x*x

if __name__ == '__main__':
    with Pool() as p:
        p.apply(f, [1])

answered Oct 08 '21 at 09:08

Could you please explain why adding a test fixed the issue ? – Philippe Oct 08 '21 at 10:46
This issue has been referred to many times in Stackoverflow. Here's an example https://stackoverflow.com/questions/20222534/python-multiprocessing-on-windows-if-name-main – Oct 08 '21 at 11:50
Thank you but this does not work on Mac. I read some similar pages before but all of them are about Windows – Qinyang Tan Oct 09 '21 at 09:17
It works with other non-interactive interpreters. Thank you. – Qinyang Tan Oct 09 '21 at 10:08
I developed and ran this code on macOS. Make sure you run it from a command line as some IDEs seem to have issues with this – Oct 09 '21 at 10:47

score 0 · Accepted Answer · answered Oct 08 '21 at 19:32

Python for MacOS changed the default "start method" in Python 3.8. This means you will now need to protect your code from execution on import by placing all multiprocessing code (and anything else that should only run in the parent process) inside a if __name__ == "__main__": block. You will also run into issues with interactive interpreters like jupyter, or IPython because there isn't always a "main" file to import. It is sometimes possible to configure your IDE to get around this somehow, but the most compatible solution is to run your code from a system terminal. Alternatively you could manually switch back to using "fork" which is faster and uses less memory in some instances, but can fall victim to deadlock when using certain threaded modules (like logging).

I tried `if __name__ == "__main__":`, it did not work. So do I have to write functions into a file and then import them? I don't think this is reasonable. — Qinyang Tan, Oct 09 '21 at 09:30

Multiprocessing example giving AttributeError on Mac

2 Answers2