1

So I have a Flask application which I've been running on Flask's built-in server, and am ready to move it to production. This application manages several child processes. To this point, I've been handling graceful shutdown using signals. In particular, one shutdown mode I've used is to have sending a SIGHUP to the flask server cause the application to propagate that signal to its children (so they can gracefully shutdown), and then let the application process shutdown.

In production, we're planning on using mod_wsgi. I've read that wsgi applications really shouldn't be handling signals.

So my question is, how should I achieve the following behavior with this setup?:

  • When apache receives SIGTERM, it notifies the wsgi daemons before terminating them
  • The wsgi daemons are given a chance to do some cleanup on their own before shutting down
augray
  • 3,043
  • 1
  • 17
  • 30

2 Answers2

2

Send SIGTERM to the Apache parent process and that is what sort of happens now.

What happens is that when the Apache parent process receives SIGTERM, it in turn sends SIGTERM to all its child worker processes, as well as to the managed mod_wsgi daemon processes if using daemon mode. Those sub process will stop accepting new requests and will be given up to 3 seconds to complete existing requests, before the sub processes are forcibly shutdown.

So the default behaviour of SIGTERM is to allow a bit of time to complete requests, but long running requests will not be allowed to hold up complete server shutdown. How long it does wait for sub processes to shutdown is not configurable and is fixed at 3 seconds.

Instead of SIGTERM, you can send a SIGWINCH signal instead. This will cause Apache to do a graceful stop, but this has issues.

What happens in the case of SIGWINCH, is that Apache will send SIGTERM to its child worker process again, but instead of forcibly killing off the processes after 3 seconds, it will allow them to run until at least any active requests have completed.

A problem with this is that there is no fail safe. If those requests never finish, there is no timeout that I know of which will see the child worker processes forcibly shutdown. As a result, your server could end up hanging on shutdown.

A second issue is that Apache will still forcibly kill off the managed mod_wsgi daemon processes after 3 seconds and there isn't (or wasn't last time looked) a way to override how Apache manages those processes, to enable a more graceful shutdown on the managed daemon processes. So graceful stop signal doesn't change anything when using daemon mode.

The closest you can get to a graceful stop, is in a front end routing layer, divert new traffic away from the Apache instance. Then through some mechanism trigger within the host running Apache a script which sends a SIGUSR2 to the mod_wsgi daemon processes. Presuming you have set graceful-timeout option on the daemon process group to some adequate failsafe, this will result in the daemon processes exiting if all active requests finish. If the timeout expires, then it will go into its normal process shutdown sequence of not accept new requests from the Apache child worker processes and after the shutdown-timeout (default 5 seconds) fires, if requests still not complete, the process is forcibly shutdown.

In this case, it isn't actually shutting down the processes, but causing them to exit, which will result in them being replaced as we aren't telling the whole Apache to stop, but just telling the mod_wsgi daemon processes to do a graceful restart. In this situation, unless you monitor the set of daemon processes and know when they have all restarted, you don't have a clear indication that they are all done and can then shutdown the whole Apache instance.

So it is a little bit fiddly to do and it is hard for any server to do this in a nice generic way as what is appropriate really also depends on the hosted application and what its requirements are.

The question is if you really need to go to these lengths. Requests will inevitably fail anyway and users have to deal with that, so often interruption of a handful of requests on a restart is not a big deal. What is so special about the application that you need to set a higher bar and attempt to ensure that zero requests are interrupted?

Graham Dumpleton
  • 57,726
  • 6
  • 119
  • 134
  • Thanks for the detailed answer! For us, it's not so much about making sure no requests get interrupted, and more about making sure the application has a chance to clean up after itself. In particular, our WSGI application spawns off child processes that are long-lived (they outlive requests, but won't outlive Apache since this whole shebang is running in a docker container), and which need to do some clean up on some persisted resources when they exit. For now, I'll start looking at other options (including possibly removing need for the cleanup). – augray Sep 29 '17 at 22:04
  • Do those extra spawned processes need to be spawned from the same process that handles the requests? There is a way of using mod_wsgi to spawn and manage separate processes for you, with them receiving a SIGTERM automatically when Apache is shutdown. And if they don't shutdown in 3 seconds they will be killed, just like daemon processes are. – Graham Dumpleton Sep 29 '17 at 22:19
  • Yes, they are spawned (and sometimes shutdown) using application requests. – augray Sep 29 '17 at 22:48
  • That would imply you must run with only one daemon process, else you would need some cross process locking scheme to ensure don't start more than one, if can only be one. – Graham Dumpleton Sep 29 '17 at 23:36
  • That (a cross-process locking scheme) we *do* have (it's rudimentary right now, but we're working on it). – augray Sep 29 '17 at 23:48
  • Another option is to use a dummy daemon process group (of a single process), which implements a little interface callable using cross process calling mechanisms in multiprocessing module, to provide a means of creating/destroying sub processes. I have example code. If interested, better to drop a message on the mod_wsgi mailing list about it. – Graham Dumpleton Sep 30 '17 at 00:10
0

Since mod_wsgi 4.8.0 you can also do the following:

import mod_wsgi

def shutdown_handler(event, **kwargs):
  # do whatever you want on shutdown here

mod_wsgi.subscribe_shutdown(shutdown_handler)
Cito
  • 5,365
  • 28
  • 30