Python multiprocessing processes instantly die after being started

Question

I am struggling with using Python "multiprocessing" module. I want to fill a queue with simple things and to print things when the queue isn't empty, which will let me know that one of my processes has terminated.

You will find a minimal example below:

1. I create an empty queue q

2. I create a list of processes processes, as many as my CPU will let me; I start every process in the list as soon as it's created (proc.start())

3. The target function of each process is a very simple function f which first waits for 2 seconds, and then writes 'hello' in the queue (which is its only parameter)

4. Afterwards, I check every 0.5 second if q isn't empty (it should receive a 'hello' every time one of my processes successfully executes function f), and I also check whether my processes are alive or not.

I'm facing 2 issues here:

• from the very first attempt, all my processes are not alive;
• nothing is ever printed, which means that queue q never successfully receives any 'hello'.

You will find my code below.

# -*- coding: utf-8 -*-
"""
Created on Fri Dec  4 12:21:23 2020

@author: rbourgeon
"""
import multiprocessing as mp
import time


def f(q):
    time.sleep(2)
    q.put('hello')


pool_size = mp.cpu_count() - 1
print(f'pool_size is {pool_size}')

q = mp.Queue()

processes = []
num_active_processes = 0

# Starting processes
while len(processes) < pool_size:
    proc = mp.Process(target=f,
                      args=(q,)
                      )   
    processes.append(proc)
    proc.start()
    print(f'{len(processes)} jobs started')
    num_active_processes += 1

# Checking if queue is empty every 0.5 second. If not empty, we pop an element
# and we print it
    
for i in range(1, 100):
    print(f'\nAttempt #{i}')
    if not q.empty():
        print(q.get())
    time.sleep(0.5)
    print(processes)
    print([p.is_alive() for p in processes])

The following gets printed to the console:

pool_size is 7
1 jobs started
2 jobs started
3 jobs started
4 jobs started
5 jobs started
6 jobs started
7 jobs started

Attempt #1
[<Process(Process-8, stopped[1])>, <Process(Process-9, stopped[1])>, <Process(Process-10, stopped[1])>, <Process(Process-11, stopped[1])>, <Process(Process-12, stopped[1])>, <Process(Process-13, stopped[1])>, <Process(Process-14, stopped[1])>]
[False, False, False, False, False, False, False]

Attempt #2
[<Process(Process-8, stopped[1])>, <Process(Process-9, stopped[1])>, <Process(Process-10, stopped[1])>, <Process(Process-11, stopped[1])>, <Process(Process-12, stopped[1])>, <Process(Process-13, stopped[1])>, <Process(Process-14, stopped[1])>]
[False, False, False, False, False, False, False]

and so on until last attempt.

which means that a) all of my processes died within 0.5 seconds b) in the meanwhile, none of them successfully executed function f because the "print" line is never executed (and therefore the queue is empty).

Answer 1

I found the reason why my code isn't working as expected: the block of code that start the processes should be encapsulated in a if __name__ == '__main__':.

The documentation of multiprocessing module says:

one should protect the “entry point” of the program by using if __name__ == '__main__':

doing so, the code works properly.

Answer 2

Your f function does not kee anything running as a worker: once it runs its last line, it ends and returns - that will shutdown the process it lives on.

So, it is expected that no process will be active in your final check.

The surprise is that you say the value does not show up in the caller queue - and as you found out, it is because in Windows, the code that is related to starting the subprocess calls itself should only run on the main process (and the way to do it is checking if the __name__ variable equals to "__main__") - and actually , it is likely that you just did not see the print. I've run your code here, and it works as expected. Note that you just print one queue element in each of your "attempts". Since each process will just put a single element in the queue and exit, you should see a "hello" printed out after 8 of the first 15 or so attempts (the first 4 attempts will take place while the subprocess are in the 2-second "sleep") , and no prints afterwrds, as the processes are finished.

If you want to keep the workers running, you should have a while loop in your target function, that will try to read messages from the queue and dispatch tasks. It is not something hard to write, but Python has that already done for you in the concurrent.futures library: Using that approach, yes, you create a subprocess pool that is kept alive by the Python runtime while you don't explicitly shut it down, and each of your target functions can be called individually from the main process anytime you want, reusing the process - and the function is just a normal Python function - no need to have a loop, listen to the queue, etc...