0

I understand that multiprocessing starts differently in linux vs Windows where linux fork() and window spawn()

spawn

The parent process starts a fresh python interpreter process. The child process will only inherit those resources necessary to run the process objects run() method. In particular, unnecessary file descriptors and handles from the parent process will not be inherited. Starting a process using this method is rather slow compared to using fork or forkserver. [Available on Unix and Windows. The default on Windows and macOS.]

fork

The parent process uses os.fork() to fork the Python interpreter. The child process, when it begins, is effectively identical to the parent process. All resources of the parent are inherited by the child process. Note that safely forking a multithreaded process is problematic. [Available on Unix only. The default on Unix.]

I am using Windows and I have a script where I am importing some personal module.

main.py that imports test.py:

import test
import multiprocessing as mp

def f(x,y):
   z = test.add(x,y)
   print(z)

if __name__ == '__main__':
   pool = mp.Pool(2)
   pool.map(f, [(x,x+1,) for x in [1,2]])

Ideally, this will spawn 2 processes. What I want to do is that make sure each of this process does not have to re-import test.py again since import test is a super heavy loading process.

I believe if I can find a way to switch Windows to use fork instead of spawn, it might resolve the issue. But I am not 100% sure.

martineau
  • 119,623
  • 25
  • 170
  • 301
Bill Nguyen
  • 29
  • 5
  • from wikipedia: *"Microsoft Windows does not support the fork-exec model, as it does not have a system call analogous to `fork()`"* while this isn't the whole story, it's functionally true. [This](https://stackoverflow.com/a/62888362/3220135) answer explains it nicely. – Aaron Jan 26 '21 at 00:57
  • What part of "Available on Unix only" was unclear? – martineau Jan 26 '21 at 01:07
  • I wasn't guessing that if there is a fork-exec model, it might solve it. But my actual question is "How I can pass library imported by parent to the child without the child processes having to imported it themselves?" If there is no easy way to fork() it which make sense. I just think, there should be a way to pass down library imported by parents to child processes. – Bill Nguyen Jan 26 '21 at 01:47

0 Answers0