Watchdog-like support (bash perl or python)

Question

I have a bash script (that I'm converting to perl) that runs in an infinite loop (while true; do) to poll devices on our network and log their responses to a text file. With each iteration of the (while) loop, the text file for that device is appended with its latest information.

I would like to have this script always run -- if it gets hung, crashes, or is no longer writing to the appropriate text files -- it should be restarted.

Following the advice posted in this StackOverflow question, I could write the following bash script:

until myserver; do
  echo "Server 'myserver' crashed with exit code $?.  Respawning.." >&2
  sleep 1
done

where myserver is the polling program. This would account for issues with the script unexpectedly crashing or hanging, presuming a non-zero exit code were issued in those cases. However, if the script doesn't completely fail/exit, but fails in a way that just stops it from writing out to the text files -- I'd like to restart the script in this case as well. This is where a watchdog-like script would come in. I could use Python's watchdog and write a script that uses Observer library to monitor the text files being generated, like in this example. I would trigger on stagnant text files to issue a non-zero exit for the python script and then augment the above bash script as follows:

until [myserver -o pythonMon]; do
  echo "Server 'myserver' crashed with exit code $?.  Respawning.." >&2
  sleep 1
done

where pythonMon is the python script monitoring whether or not the text files are updating properly. Unfortunately, this approach requires three scripts (the main polling script and two monitoring scripts); it's a bit of a kludge . I'm looking to optimize/simplify this approach. Any recommendations? Ideally, I'd have a single script (at least a single monitoring script) to keep the polling script running versus two. Would there be a way to add file monitoring directly into the bash or perl code? This is running on 64-bit, CentOS 6.5

Answer 1

I'm doing something rather similar for monitoring a bunch of devices. Depends a bit on polling frequency though - I'm spawning via cron, at 3m intervals.

Bear in mind 10s samples are potentially quite intensive, and may not always be necessary - it does depend a bit on what you're aiming for.

Anyway, the tool for the job is Parallel::ForkManager.

#!/usr/bin/perl

use strict;
use warnings;

use Parallel::ForkManager;

my @targets = qw( server1 server2 );

my %test_list = { 'fetch_cpu' => \&fetch_cpu_stats, };


sub fetch_cpu_stats {
    my ($target) = @_;
    ## do something to $target;
    open( my $ssh_results, "-|", "ssh -n $target uptime" )
        or die $!;
    while (<$ssh_results>) {
        print;
    }
}

my $manager = Parallel::ForkManager->new(10);

while (1) {
    foreach my $test ( keys %test_list ) {
        foreach my $target (@targets) {
            $manager->start and next;
            print "$$ starting $test\n";
            &{$test_list{$test}}($target);
            $manager -> finish;
        }
    }
    sleep 10;
}

This'll spawn up to 10 concurrent 'tests', and re-run them every 10s. It's probably worth some sort of 'lock' process (using flock) to make it easy to use cron check if your 'daemon' script is still running.

That would be something like:

open ( my $self, "<", $0 ) or die $!;
flock ( $self, 2 | 4 ) or die "$0 already running"; 

You can then fire it in cron every so often, and it'll restart itself if it's died for some reason.

But anyway - you can have multiple subroutines (e.g. your test scripts) all spawned autonomously (and for bonus points - they'll run in parallel).