How to guard Python multiprocessing on Windows from unwanted recursion

Question

So, I'm doing this project in which the following steps are ran (necessarily as they are):

Optimization algorithm program (package O) calls aircraft analysis script as a subprocess;
Aircraft analysis script (script S) calls an aerodynamics package;
The aircraft analysis package (lets call it package P) creates several multiprocessing Process class instances to parallelize heavy tasks.

On Linux, this works without any issue.

On Windows, however, package P needs a guard stopping the child processes from recursively re-importing their father script S (as in python multiprocessing on Windows), which leads to an infinitely repeated task execution.

It isn't hard to implement it adding a guard to script S:

if __name__=='__main__':
    #do stuff, call methods from package P

I didn't want my users to have to use that guard in every call to the package, however, so I've added a guard to package P instead. It is implemented as a flag, as in (names edited for simplicity, the package has over 5000 lines):

class FirstClass:
    #first method in package P to be executed in script S is FirstClass.__init__
    def __init__(self):
        self.ischild=(multiprocessing.current_process().name != 'MainProcess')
    def some_other_method(self):
        if not self.ischild:
            do_stuff_that_the_package_should_do()

So, essentially, package P's funcions and methods are only ran if flag ischild is set to False in the first class instantiated by script S.

It works when script S is called from command line, and stops script S from doing anything in a child process as it is re-imported - therefore stopping both redundant tasks and recursive subprocess creation.

When I try to configure optimization package O to run script S as a subprocess, however, this guard stops working, because script S is no longer the main process. Flag ischild is, then, set to True even when script S is ran for the first time by package O, because multiprocessing.current_process().name is no longer equal to 'MainProcess' even if the call is not recursive.

Is there any other way I can implement a guard to package P to stop its recursive calls to the main process that would work even if script S is not the main process?

It would be nice if I could identify exactly which package/script summoned the process, with something like:

if multiprocessing.father()=='package_P':
    ischild=True #stops execution of function do_stuff_the_package_should_do() when the call is recursive

So that no trouble is caused when the calling process is neither the main process (as it wouldn't be if script S is a subprocess of another script or package) nor a child process from package P (recursive call).

score 1 · Accepted Answer · answered Apr 17 '20 at 18:02

Just refactor your code in a way that there's no implicit automagical multiprocessing happening when S is imported.

e.g.

S/
  __init__.py
  __main__.py

where only S/__main__.py starts whatever you're doing. You can then run python -m S and Python will invoke that main module. Simply importing S (or anything from S) won't do that.

How to guard Python multiprocessing on Windows from unwanted recursion

1 Answers1