4

I have a python program that is running a number of worker processes. Because this has to be handled properly to avoid orphaned processes, I have implemented a signal handler to shutdown all worker processes.

The program starts more or less like this:

  1. Start process pool (starts X number of workers)
  2. Register signal handlers (signal.signal(signal.SIGTERM, my_signal_handler)). I also add another signal handler for SIGINT with the same handler.
  3. Start seperate thread polling backend (database) and add the tasks to process pool.
  4. On main thread, poll process pool for results (there is a result multiprocessing.Queue that the individual workers add the results to).

The idea is that the two seperate threads started in 3 and 4 keep the tasks running through the machinery.

If I start this manually and call kill -15 <pid> or kill -2 <pid> it correctly shuts everything down, waits for processes to join(). Reading from the documentation, runit sends a TERM to the process, followed by CONT. However, running this under runit, it simply shows the standard ok: down: <my_program>: 1s, normally up, but the process is still running in the background (even the main process, it is UNTOUCHED).

If I then afterwards go out and manually kill the process, I can see in the log file that it shuts down correctly. What am I doing wrong? It seems that runit ONLY kills the 3-line shell script I created to activate the virtualenv, but leaves the actual python process behind.

Even if I run the "run" script directly, I can either run kill or Ctrl+C (same as SIGINT) and it shuts down correctly.

Christian P.
  • 4,784
  • 7
  • 53
  • 70
  • Have you considered using a different mechanism to run, control and monitor the workers? Supervisord, for example. It might not have the same behavior problems... – Charles Aug 13 '13 at 16:01
  • I have, but as it is we have a lot of other services under runit, so it would be annoying to have two "competing" tools. – Christian P. Aug 14 '13 at 09:42
  • @Charles, `runit` can have custom hooks written to replace its default signaling behavior, so it can do literally anything supervisord or kin can. – Charles Duffy Aug 14 '13 at 12:30

1 Answers1

3

Okay, so after some extensive testing I figured it out.

Runit will send the kill signal to the run script, which does not propogate it by default. What you need to make sure is that you call exec python yourscript.py in the end. Similarly, if your run script calls another shell script (ie. one that activates your virtualenv or similar), it must do so with exec as well.

Samples:

run:

#!/bin/sh
umask 002
2>&1
exec chpst -uanalytics cliscript router

cliscript:

#!/bin/sh

# Resolve script path, assuming that the script resides in $(ABSPATH)/bin
SCRIPTPATH="$0"
if [ -h "$SCRIPTPATH" ]; then
    SCRIPTPATH=$(readlink -e "$0")
fi
ABSPATH=$(dirname "$(cd "$(dirname "$SCRIPTPATH")"; pwd -L)")

# Load the virtual environment
source "$ABSPATH/venv/bin/activate"

# Set up environment
export PYTHONUNBUFFERED=1

exec python "$ABSPATH/bin/processing-cli.py" $@

Take note of the exec being called when we "pass" control to the next script or python itself.

Christian P.
  • 4,784
  • 7
  • 53
  • 70
  • Your question didn't contain enough information to make this diagnosis, by the way -- one could reasonably assume that you'd read the sample run scripts, all of which correctly use `exec`, before writing your own, and the question didn't an actual sample of the failing runscript by which to disabuse someone of this notion. – Charles Duffy Aug 14 '13 at 12:32
  • 1
    ...also, `$@` needs to be `"$@"` if you don't want your arguments to be string-split. (Effectively, unquoted `$@` is exactly the same as `$*`). – Charles Duffy Aug 14 '13 at 12:32
  • 1
    I know, which is also why I put the answer here. I did not think it would be relevant to include the scripts - it's only after I found out that exec was important that it would've made sense to include them, and by then I knew what was wrong :) – Christian P. Aug 15 '13 at 09:37