121

I am having a service that depends on Cassandra coming up gracefully and the cluster being up and ready.

To ensure that the dependency order is met, I have the following unit file 

[Unit]
Requires=cassandra.service
After=cassandra.service

[Service]
Environment=JAVA_HOME=/usr/java/jre
ExecStart=@bringup.instance.path@/webapps/bringup-app/bin/bringup
TimeoutStartSec=0
ExecStop=
PIDFile=@bringup.instance.path@/logs/bringup.pid
Restart=always

[Install]
WantedBy=multi-user.target

How do I ensure that the bringup-app process waits for 30 seconds before it attempts to start up? Currently although it is started after Cassandra, I have noticed that the Cassandra cluster is not up yet and hence any attempt from the bringup-app to connect to Cassandra as part of startup fails.

I therefore want to add a delay. Is that possible via the unit file?

Ortomala Lokni
  • 56,620
  • 24
  • 188
  • 240
Sirish Renukumar
  • 1,509
  • 2
  • 13
  • 16
  • cassandra-service should only return once fully started. That is, the launcher should wait until the service is ready, then exit. Also, cassandra-service could make use of socket activation. – André Werlang Jul 28 '20 at 22:43

7 Answers7

193

You can run the sleep command before your ExecStart with ExecStartPre :

[Service]
ExecStartPre=/bin/sleep 30
Ortomala Lokni
  • 56,620
  • 24
  • 188
  • 240
  • 29
    how do you solve this without having a sleep on service restart? – rubo77 Aug 14 '18 at 19:47
  • 12
    The fact that this works on a service restart is a bonus for my use case – sergtech Dec 08 '18 at 22:52
  • Don't know why but it makes a 30sec loop for the service and `ExecStart` not going to be started. – M. Rostami Feb 23 '20 at 06:32
  • 5
    I feel like systemd doesn't really want me to do this. With a longer delay this causes timeouts and e.g. `systemctl start x` doesn't return until the sleep has completed – phiresky Feb 23 '20 at 23:14
  • I too would like to see a more-elegant dependency "connection," something in the spirit of `WaitOn=cassandra.service.output.deliverable`; this is possible what [this answer](https://stackoverflow.com/a/56957751/605356) tries to accomplish? – Johnny Utahh Jul 18 '20 at 18:14
  • 7
    When using the above method with delays longer than 90 seconds, the service’s start timeout will have to be increased (e.g. `TimeoutStartSec=120`) – SomeDude Jul 03 '21 at 06:23
70

You can create a .timer systemd unit file to control the execution of your .service unit file.

So for example, to wait for 1 minute after boot-up before starting your foo.service, create a foo.timer file in the same directory with the contents:

[Timer]
OnBootSec=1min

It is important that the service is disabled (so it doesn't start at boot), and the timer enabled, for all this to work (thanks to user tride for this):

systemctl disable foo.service
systemctl enable foo.timer

You can find quite a few more options and all information needed here: https://wiki.archlinux.org/index.php/Systemd/Timers

mj3c
  • 1,447
  • 15
  • 18
  • 3
    While this will work, it is an overkill for a simple delay. For simple delay use ExecStartPre and use timers for more complex schedules. – mikijov May 22 '19 at 17:44
  • 2
    OnBootSec is apparently relative to when the box "first started" so if the service you are trying to "start after" doesn't happen to startup within 1m, it could still fire too early, I wonder? https://www.freedesktop.org/software/systemd/man/systemd.timer.html But I guess if you set a long enough number it could work... – rogerdpack Jul 09 '19 at 16:31
  • @rogerdpack That is true. However with Cassandra, even after the service starts, starting a session (with cqlsh or otherwise) is not instantly possible, which is why this approach is more convenient for this specific case. But yeah you are basically "guessing" the time needed for it to become available. – mj3c Jul 10 '19 at 08:22
  • 1
    IIRC you also need to start the timer `systemctl start foo.timer`. – Robert Klemme Jan 16 '20 at 13:17
  • 5
    IMO This is the best answer because systemd considers a unit to be 'starting' when the ExecStartPre directives are executed. – Paul Back Mar 30 '20 at 21:56
35

Instead of editing the bringup service, add a post-start delay to the service which it depends on. Edit cassandra.service like so:

ExecStartPost=/bin/sleep 30

This way the added sleep shouldn't slow down restarts of starting services that depend on it (though does slow down its own start, maybe that's desirable?).

rogerdpack
  • 62,887
  • 36
  • 269
  • 388
bviktor
  • 1,356
  • 13
  • 14
30

Combining the answers from @Ortomala Lokni and @rogerdpack, another alternative is to have the dependent service monitor when the first one has started / done the thing you're waiting for.

For example, here's how I am making the fail2ban service wait for Docker to open port 443 (so that fail2ban's iptables entries take priority over Docker's):

[Service]
ExecStartPre=/bin/bash -c '(while ! nc -z -v -w1 localhost 443 2>/dev/null; do echo "Waiting for port 443 to open..."; sleep 2; done); sleep 2'

Simply replace nc -z -v -w1 localhost 443 with a command that fails (non-zero exit code) while the first service is starting and succeeds once it is up.

For the Cassandra case, the ideal would be a command that only returns 0 when the cluster is available.

(May want to increase TimeoutStartSec from the default 90s as well, or set TimeoutStartSec=0 to disable startup timeouts)

PolyTekPatrick
  • 3,122
  • 1
  • 25
  • 19
Graham Lea
  • 5,797
  • 3
  • 40
  • 55
  • 5
    Best answer on here and completely underrated! I added the note from rogerdpack's answer about increasing TimeoutStartSec, or you can set it to 0 to disable startup timeouts. – PolyTekPatrick Mar 18 '21 at 21:20
  • 1
    The brilliance of this is that it can be adapted to suit virtually any need, and since it doesn't have to wait for a fixed amount of time before the boot sequence can continue, it's a good method to delay the boot process, especially if the length of the delay is unknown. – Robidu Oct 08 '22 at 18:41
  • This was an inspirational answer! I added this to my service; I hate arbitrarily long delays. I want the thing to get rolling as soon as it can! This was for mounting a Veritas filesystem: `ExecStartPre=/bin/bash -c 'delay=60; i=0; while [ $i -lt $delay ]; do printf "$i "; [ -e /dev/vx/dsk/veritas0 ] && break; let i=$i+1; sleep 1; done; if [ $i = $delay ] ; then echo "ERROR: Timeout; exited."; exit 1; else exit 0; fi'` – Mike S Nov 02 '22 at 20:16
  • Note that you shouldn't have to put the parers around the while loop. I haven't tested it, but my example doesn't use them. They'll just add an additional subshell which is not necessary in this case. – Mike S Nov 03 '22 at 13:36
9

This answer on super user I think is a better answer. From https://superuser.com/a/573761/67952

"But since you asked for a way without using Before and After, you can use:

Type=idle

which as man systemd.service explains

Behavior of idle is very similar to simple; however, actual execution of the service program is delayed until all active jobs are dispatched. This may be used to avoid interleaving of output of shell services with the status output on the console. Note that this type is useful only to improve console output, it is not useful as a general unit ordering tool, and the effect of this service type is subject to a 5s time-out, after which the service program is invoked anyway. "

rogerdpack
  • 62,887
  • 36
  • 269
  • 388
nelaaro
  • 3,006
  • 5
  • 38
  • 56
  • It is very similar to oneshot too, simple won't run ExecStop for commands that exit shell after execution like vncservers (of course that can be tweaked to not run in background) but i mean that it runs also for background processes. – m3nda Mar 31 '18 at 01:18
  • 4
    You conveniently left out the most important part from the quote: "this type is useful only to improve console output, it is not useful as a general unit ordering tool, and the effect of this service type is subject to a 5s timeout, after which the service program is invoked anyway [...] it is generally not recommended to use idle or oneshot for long-running services." – bviktor Feb 18 '19 at 19:06
  • 2
    @bviktor, the documentation has updated. You were correct, it did not explain its purpose correctly The documentation does say `however, actual execution of the service program is delayed until all active jobs are dispatched`. The documentation seams to contradict itself. – nelaaro Feb 19 '19 at 13:08
  • This still might not be enough if your service starts and then immediately the boot sequence "ends and enters idle" there might not be a long enough delay... :| – rogerdpack Jul 09 '19 at 16:25
  • I doubt that this is going to work, because you are provoking a race condition. Expect any jobs that have already been scheduled to be completed before the one that you have marked as idle so there's a high chance that whatever you are attempting to perform after your checks to be executed before them. Your best bet still would be to directly check for the required service's availability, e. g. by looking out for their control sockets or whatever may be feasible. – Robidu Oct 08 '22 at 18:37
9

The systemd way to do this is to have the process "talk back" when it's setup somehow, like by opening a socket or sending a notification (or a parent script exiting). Which is of course not always straight-forward especially with third party stuff :|

You might be able to do something inline like

ExecStart=/bin/bash -c '/bin/start_cassandra &; do_bash_loop_waiting_for_it_to_come_up_here'

or a script that does the same. Or put do_bash_loop_waiting_for_it_to_come_up_here in an ExecStartPost

Or create a helper .service that waits for it to come up, so the helper service depends on cassandra, and waits for it to come up, then your other process can depend on the helper service.

(May want to increase TimeoutStartSec from the default 90s as well)

rogerdpack
  • 62,887
  • 36
  • 269
  • 388
8

I have used systemd timer for delay a service and it work very well

# /lib/systemd/system/foo.timer 
[Unit]
Description=Wait some second before run foo

[Timer]
OnActiveSec=5sec
AccuracySec=1s

[Install]
WantedBy=timers.target

see timers : systemctl list-timers

log:

journalctl -f -u foo.timer
journalctl -f -u foo
A.L
  • 10,259
  • 10
  • 67
  • 98
Simone Cianni
  • 81
  • 1
  • 4
  • 1
    Best way to delay executing a program on boot but not on restarts – Oriol Vilaseca Mar 29 '23 at 09:05
  • All I know is the above, for me, ran when foo started, not before foo started. I.e., it did not delay foo one single bit. – Dan Jacobson Jun 30 '23 at 13:40
  • https://github.com/systemd/systemd/issues/24026#issuecomment-1615867761 says to use the "edit" command instead. That creates a /etc/systemd/system/foo.service.d/override.conf . Nifty! – Dan Jacobson Jul 01 '23 at 12:59