Using multiprocessing.join() is not blocking execution

Question

Here is a bit of code I've run. I'm trying to make sure that all my processes finish before I move on through my code, but it's not happening as I expect.

import sys
import multiprocessing as mp
import time
import random

def testJoin(argin):
    name =  mp.current_process().name
    exStartTime = time.time()
    time.sleep(argin * random.random())
    print(name + ' took %d seconds' %(time.time() - exStartTime))
    sys.stdout.flush()


if __name__ == '__main__':
    instances = [10, 10, 10, 10]
    jobs = []
    for k in instances:
        p = mp.Process(target = testJoin, args = (k,))
        jobs.append(p)
        p.start()
    p.join()

print('End of Program')

Here is what's being output:

End of Program
Process-4 took 1 seconds
End of Program
End of Program
Process-2 took 4 seconds
End of Program
Process-1 took 9 seconds
End of Program
Process-3 took 9 seconds

Where I'm confused is that I don't expect to see "End of Program" printed more than once, and I certainly don't expect to see it printed until all of my four processes have ended. What am I missing?

score 0 · Answer 1 · edited May 23 '17 at 11:58

0

The behavior you describe does not happen under Unix, but it does under Windows. Windows lacks os.fork, so in order to start a subprocess, multiprocessing starts a new Python interpreter and imports the calling module.

The if __name__ == "__main__" protects code from getting executed when imported. Since the

print('End of Program')

is not inside the if-statement, it gets executed once for each time the calling module is imported and once by the main process.

The solution is to simply put the print call inside the if-statement.

edited May 23 '17 at 11:58

Community

1
1

answered Jul 09 '15 at 22:27

unutbu

842,883
184
1,785
1,677

This is a great explanation. That said, I've got what may be a stupid follow-up question. Does this mean that if have various steps in a program that depend on each other, I'll end up with nested if __name__ == '__main__' statements? Is there not a better way? – Marc Jul 23 '15 at 18:51
Usually a program is structured to have just one `if __name__ == '__main__'` near the end of the script. It could even contain a single call to a function, say, `main()`. Above the `if-statement` are the lines to import modules, define global constants, function definitions. I don't see any reason to ever nest `if __name__ == '__main__'` statements, since the inner `if-statements` conditional would always be True anyway... – unutbu Jul 23 '15 at 18:57
You can think of the statements in the `main` function as those to be executed by the main process, while those in the `testjoin` function to be those executed by the spawned subprocess(es). Any interaction between the two has to be done through Queues, or shared memory data structures, or some other mechanism for interprocess communication (IPC). – unutbu Jul 23 '15 at 19:02

Using multiprocessing.join() is not blocking execution

1 Answers1