Issue with spawning multiple processes

Question

I am running a code and getting this error:

RuntimeError: An attempt has been made to start a new process before the current process has finished its bootstrapping phase.

    This probably means that you are not using fork to start your
    child processes and you have forgotten to use the proper idiom
    in the main module:

Could someone help me with this? Here is the code, I am running:

def heavy(n, myid):
        for x in range(1, n):
            for y in range(1, n):
                x**y
        print(myid, "is done")
    
    
    start=time.perf_counter()
    
    big=[]
    for i in range(50):
        p=multiprocessing.Process(target=heavy,args=(500,i))
        big.append(p)
        p.start()
    
    for i in big:
        i.join()
    
    end=time.perf_counter()
    
    print(end-start)

Are you on windows? Did you use the `if __name__ == '__main__':` guard? — azelcer, Feb 07 '22 at 11:51
No, I am on mac. I didnt use it. But on using it, I was able to avoid the error, though I am not sure why. — akash tk, Feb 08 '22 at 11:39

score 0 · Answer 1 · answered Feb 08 '22 at 15:18

Depending on the OS, the new process will be either forked from the current one (in linux) or spawned (Windows, mac). A simplistic view of forking is that a copy of the current process is made: all the objects already created will exists in the new copy, alas will not be the same objects as in the initial process. Indeed, objects in one process are not directly accessible from the other process. Spawning starts a new python interpreter, re-runs the file (with a __name__ different from '__main__'), and then runs the target passed when creating the Process. The re-running of the file is what is part of the 'bootstraping phase'.

In your case, as you do not use the if __name__ == '__main__': guard, when the file is re-run tries to start new process, which would lead to an infinite tree of new processes. The multiprocessing module takes care of this cases and detects that you are trying to start new processes even before the target was run.

For a simple test of what happens when you use multiprocessing, see the code in this answer.

Issue with spawning multiple processes

1 Answers1

Linked