Avoid multiprocessing.Pool to run Globals

Question

I am currently working in Python. I need to work with multiprocessing.Pool and set the current working directory as the file dir an then adding the parent directory to sys.path. The problem is that, when multiprocessing.Pool starts, it runs again the line sys.path.insert(0,'..') which I want to avoid. Is there any way to make multiprocessing.Pool not running the Globals or anything which is outside the if __name__ == "__main__": snippet of code?

import os
import sys
import multiprocessing

os.chdir(sys.path[0])
sys.path.insert(0,'..')
print(os.getcwd())

def Worker(j):
    pass

if __name__ == "__main__":
    with multiprocessing.Pool(1) as p:
        p.map(Worker, range(10))
        p.close()
        p.join()

that is what `if __name__ == "__main__":` is for, you put code that is supposed to run only in the parent process under the `if` — Z Li, Apr 26 '22 at 18:50
@ZLi Of course, indeed that is what I was expecting. But, at least for my example, is not the case. — BloomShell, Apr 26 '22 at 18:52
This sounds like it could be an XY problem -- why do you need to add different things to your path? Could you instead just open the files directly without modifying path? — Dennis, Apr 26 '22 at 22:30

score 0 · Accepted Answer · answered Apr 26 '22 at 19:55

So the way multiprocessing works is that when it hits the p.map(Worker, range(10)) line, it would start a new child process, and run the whole script again from line 1 (new child process would import the calling module). The lines under the if would not run again in child process. To avoid the sys.path.insert(0,'..') to be repeated in the child process, you need to put that line under the if.

See this for more details.

import os
import sys
import multiprocessing


def Worker(j):
    pass

if __name__ == "__main__":
    os.chdir(sys.path[0])
    sys.path.insert(0,'..')
    print(os.getcwd())

    with multiprocessing.Pool(1) as p:
        p.map(Worker, range(10))
        p.close()
        p.join()

Avoid multiprocessing.Pool to run Globals

1 Answers1