2

I am spawning 3 processes using multiprocessing.process and waiting for them to complete. If one of them fails then i want to stop all other processes and also the main program. But when i use sys.exit the execution is only stopping just the process and not the main program. Here is the snippet of the code.

proc1=process(function1)
proc2=process(function2)
proc3=process(function3)
proc1.start
proc2.start
proc3.start
proc1.join
proc2.join
proc3.join

. . .

I am running some tasks in functions 1,2 and 3. I have a condition in each function to check the returncode of the task and if the returncode is not success then i would like to stop proc1,proc2 and proc3 and stop execution of the main program. When i execute sys.exit inside the function it just comes out of that process and not the main program.

Vinod kumar
  • 21
  • 1
  • 3

2 Answers2

2

For this to work you need to have communication between the worker processes and the main process. Probably the simplest way is to use multiprocessing.Event.

Before starting the processes, create a pair of multiprocessing.Event. Give them meaningful names like stop_main and stop_workers. For portability, one should give add these Events to the arguments given for the Process target.

A worker process should call stop_main.set() when it wants the main program to exit. A worker process should also call stop_workers.is_set() regularly and exit when this returns True.

After the main process starts all the workers it should keep polling stop_main.is_set(). When that returns True it should call stop_workers.set(), join the workers and exit.

Updated:

Edited to make it shorter and hopefully make it work on ms-windows.

An example:

import multiprocessing as mp
import time


def worker(num, sw, sm):
    if num == 5:
        print('This is worker', num)
        time.sleep(1)
        print('Worker', num, 'signalling main program to quit')
        sm.set()
    while not sw.is_set():
        print('This is worker', num)
        time.sleep(0.7)
    else:
        print('Worker', num, 'signing off..')


if __name__ == '__main__':
    stop_worker = mp.Event()
    stop_main = mp.Event()

    workers = [mp.Process(target=worker, args=(n, stop_worker, stop_main))
               for n in range(1, 6)]
    for w in workers:
        w.start()
    while not stop_main.is_set():
        time.sleep(1)
    print('MAIN: Received stop event')
    print('MAIN: Sending stop event to workers')
    stop_worker.set()
    for c, w in enumerate(workers, start=1):
        w.join()
        print('worker', c, 'joined')

It runs like this:

This is worker 1
This is worker 2
This is worker 3
This is worker 4
This is worker 5
This is worker 2
This is worker 3
This is worker 1
This is worker 4
Worker 5 signalling main program to quit
This is worker 5
This is worker 2
This is worker 3
This is worker 1
This is worker 4
This is worker 5
MAIN: Received stop event
MAIN: Sending stop event to workers
Worker 3 signing off..
Worker 1 signing off..
Worker 2 signing off..
worker 1 joined
worker 2 joined
worker 3 joined
Worker 4 signing off..
worker 4 joined
Worker 5 signing off..
worker 5 joined
Roland Smith
  • 42,427
  • 3
  • 64
  • 94
  • dep_event = Event() proc1=process(function1, args= dep_event) proc1.start while True if dep_event.is_set() == "True": sys.exit(1) .... ..... .... .... function1(depp_event) .... ... .... depp_event.set() I tested the above code and still it doesn't end the main program – Vinod kumar May 04 '16 at 19:36
  • I am able to able to fix the issue and now able to stop the main program using the events. Thanks a lot for your timely help. But i can creating a subprocess(shell = True enabled) from this process, So if i terminate this process it is not killing the subprocess , So how do i make sure that i kill both the process and subprocess before exiting the main program. – Vinod kumar May 09 '16 at 16:13
-1

Here's a solution. You lose all communication to the launched process once __main__ exits, but that's ok for my use-case with matplotlib. Just close the window or, as you mentioned, close the shell.

The example script is located in detail here.

Marc Compere
  • 290
  • 4
  • 17