python multiprocessing on Windows

Question

I'm fairly new to python programming and need some help understanding the python interpreter flow, especially in the case of multiprocessing. Please note that I'm running python 3.7.1 on Windows 10. Here is my simple experimental code and the output.

import multiprocessing
import time


def calc_square(numbers, q):
    for n in numbers:
        q.put(n*n)
        time.sleep(0.2)

    q.put(-1)
    print('Exiting function')


print('Now in the main code. Process name is: ' + __name__)

if __name__ == '__main__':
    numbers = [2, 3, 4, 5]
    q = multiprocessing.Queue()
    p = multiprocessing.Process(target=calc_square, args=(numbers, q))

    p.start()

    while True:
        if q.empty() is False:
            nq = q.get()
            print('Message is:' + str(nq))
            if nq == -1:
                break
    print('Done')

    p.join()

Program Output:

Now in the main code. Process name is: __main__
Now in the main code. Process name is: __mp_main__
Message is:4
Message is:9
Message is:16
Message is:25
Exiting function
Message is:-1
Done

Process finished with exit code 0

I've seen code examples that do not contain the if name qualifier. I first tried running one of those and got several errors that pointed to trying to trying to start a process before process init had completed. According to the author of that code, it works on linux because it forks, while Windows does not (I learned later that Windows uses spawn as the process start method.).

My question centers around why the process seems to execute instructions that are outside of the target. I was surprised to see the 2nd line of the output which printed before the target sent a message to the parent. Why does the sub-process interpreter run that code that is outside of the target? Is it really running that code before the target code? Why?

Mark Tolonen · Answer 1 · 2018-12-25T16:47:38.927

11

On Windows, the new processes import the main script, which executes it. This is why, on Windows, you need if __name__ == '__main__': ... it prevents code under that if from running in every process.

Your Now in main code. line is outside the if, so it will run in every process. Move it inside the if:

import multiprocessing
import time

def calc_square(numbers, q):
    for n in numbers:
        q.put(n*n)
        time.sleep(0.2)

    q.put(-1)
    print('Exiting function')

if __name__ == '__main__':
    print('Now in the main code. Process name is:', __name__)
    numbers = [2, 3, 4, 5]
    q = multiprocessing.Queue()
    p = multiprocessing.Process(target=calc_square, args=(numbers, q))
    p.start()

    while True:
        nq = q.get()
        print('Message is:', nq)
        if nq == -1:
            break

    print('Done')
    p.join()

edited Dec 25 '18 at 16:47

answered Dec 25 '18 at 16:41

Mark Tolonen

166,664
26
169
251

Thank you for your explanation. I had the "in the main code " print outside of the "if" intentionally to confirm that the code was actually being executed by the child process interpreter, and when (before the target was executed). I see that with the dynamic nature of python, that a fresh interpreter would need to execute the code before the target would be in the correct environment. – Dec 25 '18 at 17:24
What if it's not possible to use the __name__ clause? For example, what if I want to call a function in multiprocessing mode from a class method? – Andrea Mar 24 '22 at 22:57

python multiprocessing on Windows

1 Answers1

Linked