Python Multiprocessing Child Process Self Termination and PickleError

Question

I'm trying to learn the multiprocessing module and decorators. I have two questions about the code below:

from time import sleep
from multiprocessing import Process

class async:
    def __init__(self, function):
        self.func = function
    def __call__(self, *args, **kwargs):
        p = Process(target = self.func, args = args, kwargs = kwargs)
        p.start()

@async
def printA():
    sleep(1)
    print("A")

def main():
    printA()
    printA()
    printA()

    #do more work

if __name__ == "__main__":
    main()

1. Right now I'm getting PicklingError: Can't pickle <function printA at 0x00000273F97FF1E0>: it's not the same object as __main__.printA
   I'm not sure how to fix this. I feel like this was working on my work unix machine, is it a Windows thing?

2. I want the child Processes to self terminate upon completion. I tried wrapping the function with sys.exit() at the end, and using that as the target for the Process, but sys.exit() doesn't seem to properly terminate the zombie processes. What would be the correct way?

I don't want to do a blocking join(), nor set daemon=True. The end goal would be someone could import this module, put @async on any function, and it would run in a separate process and then terminate itself when it reaches the end of the function.

Answer 1

1. Pickling Error: I could reproduce the issue. On Linux it works just fine, on Windows it doesn't. The issue seems to be the decorator which cannot be pickled correctly on Windows. Removing the decorator results in this working code (of course that's not what you want, but might convince you to try out a different approach of your design). There's also this thread on SO with the exact same problem.

   from time import sleep
   from multiprocessing import Process
   
   def printA():
       sleep(1)
       print("A")
   
   def main():
       for i in range(3):
           p = Process(target = printA)
           p.start()
   
       #do more work
   
   if __name__ == "__main__":
       main()
   

2. Zombie processes: To avoid zombie processes you need to have the parent read its childrens exit codes, you need to run join() in the parent on the child processes to do that. You could do that in the decorator like this:

   class async:
       def __init__(self, function):
           self.func = function
           self.pool = []
       def __call__(self, *args, **kwargs):
           p = Process(target = self.func, args = args, kwargs = kwargs)
           self.pool.append(p)
           p.start()
       def __del__(self):
           for p in self.pool:
               p.join()
   

This uses the effect that python does only one instance of @async, thus you can gather all the processes in self.pool. Note, that this is only run at the exit of the main process, as python would not drop the async instance earlier.