33

Why does the multiprocessing module need to call a specific function to work when being "frozen" to produce a windows executable?

Voo
  • 29,040
  • 11
  • 82
  • 156
  • Without checking into it too much, I'd guess that it's because multiprocessing needs to know about the python interpreter it's running in so that it can invoke more of them, and because packaging like py2exe do unusual things to the modules included which also need to be accounted for to import them. – Colin Valliant May 14 '13 at 07:53
  • I appreciate your extremely helpful comment on my answer regarding comparators! – GhostCat Apr 01 '19 at 17:44

1 Answers1

34

The reason is lack of fork() on Windows (which is not entirely true). Because of this, on Windows the fork is simulated by creating a new process in which code, which on Linux is being run in child process, is being run. As the code is to be run in technically unrelated process, it has to be delivered there before it can be run. The way it's being delivered is first it's being pickled and then sent through the pipe from the original process to the new one. In addition this new process is being informed it has to run the code passed by pipe, by passing --multiprocessing-fork command line argument to it. If you take a look at implementation of freeze_support() function its task is to check if the process it's being run in is supposed to run code passed by pipe or not.

Community
  • 1
  • 1
Piotr Dobrogost
  • 41,292
  • 40
  • 236
  • 366
  • 1
    One thing which is not 100% clear is the object/method which prepares the new process to be run must call freeze_support()? And that could be anywhere within the object/method but before the Process.start() method is called? – Har Feb 18 '15 at 11:06
  • I would like to get an answer to this question posed by @Har as well, if you know it. The PyInstaller recipe calls for invoking that outside of one's custom the Process class. Is that needed? Can I not just put that freeze_support() in that object's init()? That would be a hell of a lot cleaner! – BuvinJ Feb 08 '17 at 01:35
  • Or, in the "static" class scope space (sorry I don't know the Python terminology for such)? This is what I'm talking about btw: https://github.com/pyinstaller/pyinstaller/wiki/Recipe-Multiprocessing – BuvinJ Feb 08 '17 at 01:38