20

I have a service on a Redhat 7.1 which I use systemctl start, stop, restart and status to control. One time the systemctl status returned active, but the application "behind" the service responded http code different from 200.

I know that I can use Monit or Nagios to check this and do the systemctl restart - but I would like to know if there exist something per default when using systemd, so that I do not need to have other tools installed.

My preferred solution would be to have my service restarted if http return code is different from 200 totally automatically without other tools than systemd itself - (and maybe with a possibility to notify a Hipchat room or send a email...)

I've tried googling the topic - without luck. Please help :-)

clausfod
  • 315
  • 1
  • 3
  • 7
  • 7
    @shellter, I disagree that this is off-topic. Healthchecks aren't a matter of concern to end-users, but are very much a development issue/question. Would you care to describe the reasoning behind the decision to close? – Charles Duffy Sep 26 '16 at 15:14
  • 1
    ...I could possibly see an argument that this is more of an ops issue (more relevant for ServerFault than StackOverflow), but I'm not convinced that that argument is a compelling one, inasmuch as crafting the functionality the OP wants requires development knowledge -- my answer provides a pointer at a specific API to use, which isn't the kind of knowledge we'd expect to be topical in a ServerFault discussion. – Charles Duffy Sep 26 '16 at 15:17
  • 1
    ...requires development knowledge *and code changes*, inasmuch as code needs to be written that calls `sd_notify()` with healthcheck results, or performs an equivalent socket write itself. – Charles Duffy Nov 21 '16 at 13:31

1 Answers1

25

The Short Answer

systemd has a native (socket-based) healthcheck method, but it's not HTTP-based. You can write a shim that polls status over HTTP and forwards it to the native mechanism, however.

The Long Answer

The Right Thing in the systemd world is to use the sd_notify socket mechanism to inform the init system when your application is fully available. Use Type=notify for your service to enable this functionality.

You can write to this socket directly using the sd_notify() call, or you can inspect the NOTIFY_SOCKET environment variable to get the name and have your own code write READY=1 to that socket when the application is returning 200s.

If you want to put this off to a separate process that polls your process over HTTP and then writes to the socket, you can do that -- ensure that NotifyAccess is set appropriately (by default, only the main process of the service is allowed to write to the socket).

Inasmuch as you're interested in detecting cases where the application fails after it was fully initialized, and triggering a restart, the sd_notify socket is appropriate in this scenario as well:

Send WATCHDOG_USEC=... to set the amount of time which is permissible between successful tests, then WATCHDOG=1 whenever you have a successful self-test; whenever no successful test is seen for the configured period, your service will be restarted.

Charles Duffy
  • 280,126
  • 43
  • 390
  • 441