3

I want to write a service that launches multiple workers that work infinitely and then quit when main process is Ctrl+C'd. However, I do not understand how to handle Ctrl+C correctly.

I have a following testing code:

import os
import multiprocessing as mp
    

def g():
    print(os.getpid())
    while True:
        pass
        
        
def main():
    with mp.Pool(1) as pool:
        try:
            s = pool.starmap(g, [[]] * 1)
        except KeyboardInterrupt:
            print('Done')


if __name__ == "__main__":
    print(os.getpid())
    main()

When I try to Ctrl+C it, I expect process(es) running g to just receive SIGTERM and silently terminate, however, I receive something like that instead:

Process ForkPoolWorker-1:
Done
Traceback (most recent call last):
  File "/usr/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap
    self.run()
  File "/usr/lib/python3.8/multiprocessing/process.py", line 108, in run
    self._target(*self._args, **self._kwargs)
  File "/usr/lib/python3.8/multiprocessing/pool.py", line 125, in worker
    result = (True, func(*args, **kwds))
  File "/usr/lib/python3.8/multiprocessing/pool.py", line 51, in starmapstar
    return list(itertools.starmap(args[0], args[1]))
  File "test.py", line 8, in g
    pass
KeyboardInterrupt

This obviously means that parent and children processes both raise KeyboardInterrupt from Ctrl+C, further suggested by tests with kill -2. Why does this happen and how to deal with it to achieve what I want?

ShadowRanger
  • 143,180
  • 12
  • 188
  • 271
  • Does this answer your question? [Keyboard Interrupts with python's multiprocessing Pool](https://stackoverflow.com/questions/1408356/keyboard-interrupts-with-pythons-multiprocessing-pool) – Charchit Agarwal Jul 13 '22 at 15:15
  • @Charchit No, the question you linked is concerned with a bug in old python version which has been fixed since then, mine is not related to that. – steam_engine Jul 14 '22 at 09:52
  • The **accepted** answer talks about the bug, the other answers are more recent and this [one](https://stackoverflow.com/a/6191991/16310741) (on the same thread) basically says the same thing as the accepted answer here – Charchit Agarwal Jul 14 '22 at 09:57

1 Answers1

3

The signal that triggers KeyboardInterrupt is delivered to the whole pool. The child worker processes treat it the same as the parent, raising KeyboardInterrupt.

The easiest solution here is:

  1. Disable the SIGINT handling in each worker on creation
  2. Ensure the parent terminates the workers when it catches KeyboardInterrupt

You can do this easily by passing an initializer function that the Pool runs in each worker before the worker begins doing work:

import signal
import multiprocessing as mp

# Use initializer to ignore SIGINT in child processes
with mp.Pool(1, initializer=signal.signal, initargs=(signal.SIGINT, signal.SIG_IGN)) as pool:
    try:
        s = pool.starmap(g, [[]] * 1)
    except KeyboardInterrupt:
        print('Done')

The initializer replaces the default SIGINT handler with one that ignores SIGINT in the children, leaving it up to the parent process to kill them. The with statement in the parent handles this automatically (exiting the with block implicitly calls pool.terminate()), so all you're responsible for is catching the KeyboardInterrupt in the parent and converting from ugly traceback to simple message.

ShadowRanger
  • 143,180
  • 12
  • 188
  • 271
  • Using initializer for ignoring `SIGINT` is elegant, I will likely use it. But why is it delivered to the whole pool? When I tested it with `kill -2 `, it simply behaved as I expected, only sending it to parent and printing `Done`. Is this `bash` behavior? – steam_engine Jul 13 '22 at 14:48
  • 3
    @steam_engine: [Per this answer](https://stackoverflow.com/a/6110213/364696), "The `SIGINT` signal is generated by the terminal line discipline, and broadcast to all processes in the terminal's *foreground process group*. Your shell has already created a new process group for the command (or command pipeline) that you ran, and told the terminal that that process group is its (the terminal's) foreground process group." Basically, yes, hitting Ctrl-C, at least on POSIX, sends `SIGINT` to the entire foreground process group, not just the parent process. – ShadowRanger Jul 13 '22 at 14:57
  • Presumably `os.setpgrp()` could work too (untested, don't quote me), if the child process `initializer` used it to detach from the parent process group, but it's UNIX-only, so if you want consistent behavior across OSes, it's not viable. – ShadowRanger Jul 13 '22 at 15:01