24

Can anybody tell me how to handle the software watchdog in Linux?

I have a program SampleApplication which runs continuously and I need to restart it if its hangs or closes unexpectedly.

I was Googling about this and found Linux has watchdog at /dev/watchdog but don’t know how to use it. Could someone help me with an example?

My question is where to I specify my application name and delay interval to restart?

Giacomo1968
  • 25,759
  • 11
  • 71
  • 103
Verve Innovation
  • 2,006
  • 6
  • 29
  • 48
  • 1
    As far as I am aware, that only makes sure the whole system is not hanging and resets if it is. I don't think you can use it just to keep a userland application alive. – Janne Jan 31 '12 at 00:39
  • I have experience of a tool in an embedded Linux application that does just what you ask, unfortunately I cannot remember its name otherwise I'd post an answer, but I do know that it works more-or-less as you have suggested in any case, so it is not at all a bad idea. Another option is to use a specific port on the localhost address 27.0.0.1, and have processes send to it and your software watchdog daemon monitor that. /dev/watchdog is for the hardware watchdog, your daemon might maintain that, so that ultimately if the daemon stops, the system resets (as a last resort). – Clifford Dec 07 '16 at 23:18

9 Answers9

18

The linux software watchdog will reboot the machine, not just restart your process.

Well this is simply not true, it is very possible to restart single or multiple processes after the watchdog signals that the systems is hanging - you can even ABORT the reboot or do a SOFT-reboot, one is able to configure "test" and "repair"-scripts / binaries which do whatever you want them to do. The busybox-version of watchdog is stripped down to a near-unusable level ... i guess the world will never know why the busybox-devs decided to abandon primary functionalities - for now, it would be best to avoid busybox at all --> the speed-improvements are nearly inexistent, the size-decrease does not compensate the huge loss of functionality. /bin/bash is rather small - recompile everything with the flag "-Os" if size matters and you're good to go - an out-of-the-box watchdog which allows for just about everything one could want.

Oh and PLEASE do NOT create your own watchdog - that'll most likely leave you with unhandled errors and make your life bad one day.

Cody Gray - on strike
  • 239,200
  • 50
  • 490
  • 574
nope
  • 181
  • 3
  • 3
    But if the watchdog signals a *system* hang, that does not identify an individual process to restart. This is not an answer to the question - monitoring of one or more processes from another is a legitimate goal, and not one achieved through /dev/watchdog. The question does not even suggest creating his own watchdog; it merely demonstrates a misunderstanding of the difference between the /dev/watchdog device and a software monitoring process (or software watchdog). – Clifford Dec 07 '16 at 23:26
  • 1
    I don't think the negative remarks on Busybox are necessary here, they are very subjective. I would leave out the part "i guess ... one could want" from your answer. Busybox is used very commonly in embedded systems, and it is normal and intentional that not all functionality of the underlying tools are supported. Users should decide whether they want the Busybox implementation or the original one, and this choice will depend from tool to tool and context to context. – patrickdepinguin Feb 06 '23 at 15:47
  • 1
    This answer in no way answers the question. – ghostly_s Mar 08 '23 at 04:20
17

Most of the Unix/Linux init programs will manage daemons for you and restart them. Look into placing your service in /etc/inittab. Or you might be using Upstart or systemd.

All of these programs run as PID 1 and it is their job to monitor and restart system processes.

From your Busybox tag I would assume you are running an embedded system. On those, the System V style init scripts with all of their shell scripts are really overkill. You should probably rip all that out and replace it with entries in /etc/inittab or upstart or systemd jobs.

Zan Lynx
  • 53,022
  • 10
  • 79
  • 131
  • Hi @Zan thanks for your inputs, I added "::respawn:/path/app" to the /etc/inittab and the app executed fine after restarting the device, the only issue is this entry is temporary and gets removed automatically upon doing a restart of the device. Is there any way to make the initialization procedure permanent. – Verve Innovation Jan 31 '12 at 04:46
  • 1
    @ITion: It isn't supposed to be temporary. You must be using an `initrd` for your `/` directory. Or possibly you have a read-only `/` and a read-write layered on top. – Zan Lynx Jan 31 '12 at 08:12
10

How about using cron? Set up a small cron job that runs every minute. Check if your application is up (using ps) and if not, restart it.

Make a tiny script like this:

#!/bin/bash
if [ ! "$(pidof myapp)" ] 
then
  /path/to/myapp &
fi

You test if "myapp" is in the process list. "!" reverses the test. If it's not there, it runs "myapp". "&" is just so it starts in the background.

Add this to cron. Depending on your system and preferences there's several ways to do it. The classical one is to use crontab. There's lots of documentation on how to specify your crontab line, but you probably want something like this:

* * * * * /path/to/the/script.sh > /dev/null

This will run your test every minute of every hour of every… You get the idea.

Janne
  • 979
  • 4
  • 13
  • Hi @Janne thanks for your answer . But the linux I am using is Busybox and it doenst have Cron . Is there anyother way. – Verve Innovation Jan 31 '12 at 01:01
  • So, an embedded system then. Exactly what is available depends on the particular linux installed on it (busybox is just an application commonly used in many small systems). If all else fails you can always write a small c app that sits in a timed loop and monitors the process list or some status file your app writes, then restarts your process if needed. – Janne Jan 31 '12 at 01:54
  • @Janne is it possible to create a tiny script which looks for the process name as the pid always changes? kind regards, robert –  Jul 21 '12 at 19:12
  • 1
    busybox does/can have cron – pstanton Oct 11 '16 at 02:23
4

Use /etc/inittab you can utilize it to start in the specific run levels and if it is killed it shall be restarted automatically

n:2345:respawn:/path/to/app

This will make it respawn in run levels 2345 you probably only need 3 and 5 but this will work fine and is built into Linux.

Kevin Panko
  • 8,356
  • 19
  • 50
  • 61
darren102
  • 2,830
  • 1
  • 22
  • 24
2

If you are using systemd there are 2 watchdogs: one for hardware (using systemd.conf or using a watchdog daemon) and one for daemons initialized as services. If systemd is your option have a look at the following: http://0pointer.de/blog/projects/watchdog.html

fabatera
  • 324
  • 2
  • 9
2

Documentation for the watchdog is here: http://linux.die.net/man/8/watchdog

But it sounds like this is not what you want. The linux software watchdog will reboot the machine, not just restart your process.

You can easily make your own watchdog. For example, you could have your program periodically write some temp file, and launch a script that checks the file once in a while and restarts your process if it hasn't updated for some time.

TJD
  • 11,800
  • 1
  • 26
  • 34
1

You can use "Monit" utility to restart and monitor your services. Simply install by issuing command `"apt-get install monit".

Akki
  • 11
  • 2
0

If anyone has arrived at this page looking for an operating system watchdog (which is not directly what the OP wanted), this is what you need:

sudo apt-get install watchdog
service watchdog status
service watchdog start

To check that it's working execute:

tail -f /var/log/syslog | grep watchdog

You should see something like:

Jul 25 22:03:35 nuc watchdog[14229]: still alive after 733 interval(s)
Jul 25 22:03:36 nuc watchdog[14229]: still alive after 734 interval(s)
Jul 25 22:03:36 nuc watchdog[14229]: still alive after 735 interval(s)
Jul 25 22:03:37 nuc watchdog[14229]: still alive after 736 interval(s)
Jul 25 22:03:37 nuc watchdog[14229]: still alive after 737 interval(s)

I hope I'm answering the question correctly. All the other answers seem to be very different.

Sridhar Sarnobat
  • 25,183
  • 12
  • 93
  • 106
-3

You can try wdog which is a utility written in c++ and linking against the Kahless_9 framework. The source code for this can be downloaded from: https://github.com/zepher999/wdog and consequently updated to suite your own needs. There is still some TODO list changes required for the future but as is this should cater for your current requirements.

The utility requires a csv file as input wherein all processes to be watched with their arguments are contained. Upon startup, the utility starts all of these processes designated in the csv file and monitors them for exit/termination whereupon it restarts the process.

Currently wdog allows for the stopping/killing of monitored processes as well as the ability to startup the utility in hot or cold mode. Hot mode allows the utility to use cached records to monitor already started processes while cold mode starting discards such cached values thereby attempting to start all processes.

The utility also has the ability to launch an instance of itself to monitor itself, thereby having a watchdog for the watchdog.

zepher
  • 1
  • 1