118

I have a simple systemd service that needs to be periodically restarted to keep its process from bugging out. Is there a configuration option for systemd services to periodically restart them? All of the Restart* options seem to pertain to restarting the service when it exits.

Timur Shtatland
  • 12,024
  • 2
  • 30
  • 47
wes
  • 7,795
  • 6
  • 31
  • 41
  • 1
    Since this question is about a tool used to administrate a server (not generic, but production), maybe the question would be even more on-topic on Server Fault. What do you think about? https://serverfault.com/help/on-topic – Valerio Bozz May 11 '22 at 10:17
  • 1
    Well I can't comment on what it was in 2015 but the `systemd` tag now says questions about writing unit files are better off on unix.SE. – wes May 11 '22 at 15:21
  • 1
    OK but the fact that the service itself has bugs and needs a fix is more for Server Fault. Mexican stalemate? – Valerio Bozz May 12 '22 at 07:51
  • ./me wished for trashcan.software.se as this is a problem that is more than just serverfault, nor unix.se, as you are basically attempting to program a fluid system, not administrating nor configuring (which would imply server/unix), but that will splitting hairs – Hvisage Mar 05 '23 at 22:41

7 Answers7

234

For systemd version >= 229, there is an option called RuntimeMaxSec, which terminates the service after it has been running for the given period of time.

e.g. To restart every 7 days:

[Service]
Restart=always
RuntimeMaxSec=7d

To me this seems more elegant than abusing Type=notify and WatchdogSec.

systemd provides a clean way to add and override directives in systemd unit files provided by vendors. Drop-In Units are described in man systemd.unit. For example, if you wanted to periodically restart the foo service provided by a package, you would create a file named /etc/systemd/system/foo.service.d/periodic-restart.conf. The contents would be as shown above. Then:

 systemctl daemon-reload
 systemctl restart foo

You can confirm that the Drop-In unit has been loaded because it will be reported in the status output:

 systemctl status

Finally, you can confirm the directive has been included by searching the systemctl show output:

 systemctl show foo.service | grep RuntimeMax

The directive reported by systemctl show will be "RuntimeMaxUSec`

gerardw
  • 5,822
  • 46
  • 39
Alex Forbes
  • 3,039
  • 2
  • 19
  • 22
  • 27
    this is only available in systemd v >= 229. Run `systemctl --version` to see your version – Hilikus Oct 01 '18 at 17:59
  • 1
    I appear to have version 229 on Ububntu 16.04.6 :D – Lamp May 07 '20 at 23:09
  • 1
    Alex Forbes, is it possible to stop a service 10min after it started and restart automatically 3 hours later ? – Jean-Élisée Yao Aug 01 '20 at 13:05
  • 10
    Can also use units in the time field: `RuntimeMaxSec=7d` – Cameron Tacklind Aug 10 '20 at 07:01
  • 1
    @Jean-ÉliséeYao [timers](https://www.freedesktop.org/software/systemd/man/systemd.timer.html) are probably what you want. – Alex Forbes Aug 11 '20 at 09:15
  • 1
    this stops that process, how to auto restart after that time out ? – Maverick Mar 26 '22 at 16:17
  • 1
    @Maverick `Restart=always` should achieve that. Check that it is not failing to start, and the start limits, as described in https://unix.stackexchange.com/questions/289629/systemd-restart-always-is-not-honored – Alex Forbes Apr 04 '22 at 12:41
  • 1
    I tried to integrate the useful comment by @Hilikus in the answer but the queue is full now. – Valerio Bozz May 11 '22 at 10:14
  • 1
    I think the `systemctl restart foo` is not necessary. When I ran `daemon-reload` it restarted for me automatically because the process's run time already exceeded the maximum. – Chad May 11 '22 at 22:55
  • 1
    Unfortunately, this solution will invoke failure handling set by `OnFailure=`, and I rather have no false positives from me intentionally restarting a service on periodic basis. – Flair Oct 11 '22 at 17:50
66

How about a crontab. Example:

30 3 * * sun /bin/systemctl restart yourService

That would restart the service named yourService at 3:30 AM each Sunday.

You may like this solution if you want something working in whatever Unix-like server (e.g. you do not want to worry about a specific systemd version, etc.).

Valerio Bozz
  • 1,176
  • 16
  • 32
Fritz Zaucker
  • 799
  • 6
  • 5
  • 6
    I can't believe I didn't think of this! This is by far the easiest solution. – jlh Dec 18 '18 at 06:43
  • 1
    @jlh how is this any better than accepted answer ? – Ciasto piekarz Jan 08 '19 at 12:23
  • 5
    @Ciastopiekarz See the comments to the accepted answer to see why that's a problematic solution. Additionally, doing it with cron allows to restart at a specific time of day. The other solution are period-based, which might drift around with time. – jlh Jan 09 '19 at 15:24
  • 1
    I see, so thats systemd version independent solution ! – Ciasto piekarz Jan 09 '19 at 18:21
  • 2
    What happend if the service was disabled or stopped? Would this re-start any manually stopped service? – Louis Loudog Trottier Jul 22 '20 at 00:41
  • 9
    @louis-loudog-trottier This will start a stopped or disabled service. If you only want it to apply for a running service, there's the `try-restart` command instead. See https://www.man7.org/linux/man-pages/man1/systemctl.1.html – JN01 Sep 08 '20 at 11:51
  • 1
    The main issue with the accepted solution is that if someone modifies (and restarts) a service outside of the regular interval, you will get a restart at an unexpected time. This restart is always going to be daily at a specific time, which may be the intention. – Siddhartha Gandhi Aug 17 '22 at 12:01
60

I saw a solution here that seemed elegant, if a bit roundabout. The key idea is to create a one-shot service triggered by a timer that restarts another service.

For the timer:

[Unit]
Description=Do something daily

[Timer]
OnCalendar=daily
Persistent=true

[Install]
WantedBy=timers.target

For the one-shot service:

[Unit]
Description=Restart service

[Service]
Type=oneshot
ExecStart=/usr/bin/systemctl try-restart my_program.service

For the one-shot service on Ubuntu 16.04 LTS:

[Unit]
Description=Restart service

[Service]
Type=oneshot
ExecStart=/bin/systemctl try-restart my_program.service

This solution lets you leverage systemd's timers, including the ability to restart the service at a particular time of day, and not just after some amount of time has elapsed.

Everett Toews
  • 10,337
  • 10
  • 44
  • 45
matmat
  • 875
  • 8
  • 11
  • 7
    I'm confused about how the daily timer calls the one-shot service here? – svandragt Apr 05 '17 at 10:47
  • 6
    I'm not sure if this is what you're asking, but, as best as I understand it, the one-shot service has a timer associated with it, and the job of this one-shot service is to restart another service. I believe this is accomplished by simply naming the service and its associated timer with the same name, i.e., `weeklyRestart.service` and `weeklyRestart.timer`. When I worked this out, I went through a _lot_ of links. Message me and I can send you a link dump. – matmat Apr 06 '17 at 17:52
  • 6
    @svandragt That's exactly how it works. A systemd timer has to have the same name as the service it's starting. – Michael Ambrose Jun 20 '17 at 21:48
  • 3
    @MichaelAmbrose Not necessarily, you can also specify another service with `Unit=`. – Chris Down Dec 10 '17 at 15:59
  • 1
    advantage of this solution: with an "ExecStartPre" you can do some checks to higher the chance that the service starts, e.g. syntax check the config. – Marco Jun 17 '18 at 06:14
  • 3
    Why does this timer have `Persistent=true`? If restarting a service is the purpose here and the timer didn't run, because the server was offline, then the service is restarted anyway. – Dominik Apr 28 '19 at 13:13
  • 1
    A related question: https://unix.stackexchange.com/questions/546945/restart-systemd-service-with-timer-unit – rfmoz Jan 20 '22 at 16:13
55

Yes, you can make your service to restart it periodically by making your service of Type=notify. Add this option in [Service] section of your service file along with Restart=always and give WatchdogSec=xx, where xx is the time period in second you want to restart your service. Here your process will be killed by systemd after xx time period and will be restarted by systemd again. for eg.

[Unit]
.
.

[Service]
Type=notify
.
.
WatchdogSec=10
Restart=always
.
.

[Install]
WantedBy= ....
Saturn
  • 966
  • 7
  • 14
  • 8
    Clever! FWIW `type=notify` is not necessary for my dumb legacy service that just bugs out after a while. Setting `WatchdogSec` with `Type=simple` works the same, though. – wes Jun 26 '15 at 14:04
  • 33
    The problem with ``WatchdogSec`` is that it will send a SIGABRT (causing a core dump) every time it restarts. Besides uncleanly terminating the process, it also could fill up your root disk with coredumps eventually. – bk0 Jan 03 '17 at 22:54
  • 3
    @wes thanks, `Type=notify` was not making the service start for me, `Type=`simple` works great ( actually without any `Type` works fine too, probably simple is the default `Type` ) – WonderLand Mar 07 '17 at 05:31
  • 1
    Don't use `Type=notify` unless the service is already systemd-aware. Such a service's unit would already be using `Type=notify`. Using this with a service that does not speak to systemd will cause failures to start. – Michael Hampton Aug 15 '18 at 16:02
  • 7
    As of ver240, there is WatchdogSignal so you can override the SIGABRT https://github.com/systemd/systemd/blob/master/NEWS – Gary Myers Nov 27 '19 at 21:31
  • @bk0 I guess there is a limit on the coredump size. I've been running my program for a few minutes but only see a fix number of coredump files. – gyuaisdfaasf May 24 '21 at 23:08
7

Wanted to comment on the

[Service]
Restart=always
RuntimeMaxSec=604800

answer above but can't w/o more points.

Comment is that this solution will invoke failure handling set by OnFailure=failure_handling.service. Since the scheduled restart isn't a real failure any logging, notifications, etc. from the failure handling service will be unwanted and probably disruptive.

An actual periodic restart would be a sensible feature for systemd, but I won't hold my breath.

E Gow
  • 92
  • 1
  • 2
  • 2
    According to the [doc](https://www.freedesktop.org/software/systemd/man/systemd.unit.html): A service unit using Restart= enters the failed state only after the start limits are reached. – Robin Daugherty Nov 27 '19 at 19:39
5

Just some alternate approaches to ultimately reach the same goal:

  • if you have control over the service implementation you could make it end voluntarily after a while, for example either plain exiting after a certain number of iterations (if applicable) or using a timeout timer with a handler sendin itself a SIGTERM/SIGKILL
  • if voluntary service ending is not feasible/practical you could have a small cron-based script killing the service process(es).
Dan Cornilescu
  • 39,470
  • 12
  • 57
  • 97
1

Copy-paste solution for systemd version >= 229:

SERVICE="systemd-resolved.service"
env SYSTEMD_EDITOR=tee sudo -E systemctl edit --system ${SERVICE} <<EOF
[Service]
Restart=always
RuntimeMaxSec=7200
EOF

sudo systemctl daemon-reload && sudo systemctl restart "${SERVICE}"

Check:

systemd-delta | grep ${SERVICE}
# NOTE: property here must be requested with "U": RuntimeMaxUSec
systemctl show ${SERVICE} --property=RuntimeMaxUSec

Rollback:

SERVICE="systemd-resolved.service"
sudo rm -r "/etc/systemd/system/${SERVICE}.d"
sudo systemctl daemon-reload
sudo systemctl restart ${SERVICE}

More elegant would be the following:

SERVICE="systemd-resolved.service"
sudo systemctl set-property ${SERVICE} RuntimeMaxSec=7200

But it does not work for some reason:

Failed to set unit properties on systemd-resolved.service: Cannot set property RuntimeMaxUSec, or unknown property.
jetnet
  • 571
  • 5
  • 8