BASH: How monitor a script for execution failure

Question

I'm using Linux to watch a script execution in order for it to be respawned when the script runs into an execution failure. Given is a simple 1-line script which should help demonstrate the problem.

Here's my script

#!/bin/bash

echo '**************************************'
echo '*          Run IRC Bot               *'
echo '**************************************'
echo '';
if [ -z "$1" ] 
    then
        echo 'Example usage: ' $0 'intelbot'
fi

until `php $1.php`; 
    do 
        echo "IRC bot '$1' crashed with the code $?.  Respawning.." >&2; 
        sleep 5 
done;

What kill option should I use to say to until, hey I want this process to be killed and I want you to get it working again!

Edit The aim here was to manually check for a script-execution failure so the IRC Bot can be re-spawned. The posted answer is very detailed so +1 to the contributor - a supervisor is indeed the best way to tackle this problem.

We don't add tags to every valid construct in every language. — chepner, Apr 18 '15 at 15:32
i see, it was valid for the post and was valid for http://stackoverflow.com/questions/696839/how-do-i-write-a-bash-script-to-restart-a-process-if-it-dies?answertab=votes#tab-top — youdsmedia, Apr 18 '15 at 16:24
BTW, the backticks are wrong. Use `until php "$1.php"`, no backticks. — Charles Duffy, Apr 18 '15 at 16:36
Sorry just to be clear, my post was marked down but I am using valid bash syntax for a purpose I was recommended to use in http://stackoverflow.com/questions/696839/how-do-i-write-a-bash-script-to-restart-a-process-if-it-dies?answertab=votes#tab-top — youdsmedia, Apr 20 '15 at 10:17
Re: your link, by the way -- lhunath is someone I very deeply respect. It's worth noting that process supervision systems mostly work the way their post describes, by launching and waiting for children. (The exceptions are systems such as systemd, which can operate in that mode, but supports following forks and other mechanisms as well). — Charles Duffy, Apr 20 '15 at 21:47

score 2 · Accepted Answer · answered Apr 18 '15 at 16:41

2

First -- don't do this at all; use a proper process supervision system to automate restarting your program for you, not a shell script. Your operating system will ship with one, be it SysV init's /etc/inittab (which, yes, will restart programs so listed when they exit if given an appropriate flag), or the more modern upstart (shipped with Ubuntu), systemd (shipped with current Fedora and Arch Linux), runit, daemontools, supervisord, launchd (shipped with MacOS X), etc.

Second: The backticks actually make your code behave in unpredictable ways; so does the lack of quotes on an expansion.

`php $1.php`

...does the following:

Substitutes the value of $1 into a string; let's say it's my * code.php.
String-splits that value; in this case, it would change it into three separate arguments: my, *, and code.php
Glob-expands those arguments; in this case, the * would be replaced with a separate argument for each file in the current directory
Runs the resulting program
Reads the output that program wrote to stdout, and runs that output as a separate command
Returns the exit status of that separate command.

Instead:

until php "$1.php"; do 
    echo "IRC bot '$1' crashed with the code $?.  Respawning.." >&2; 
    sleep 5 
done;

Now, the exit status returned by PHP when it receives a SIGTERM is something that can be controlled by PHP's signal handler -- unless you tell us how your PHP code is written, only codes which can't be handled (such as SIGKILL) will behave in a manner that's entirely consistent, and because they can't be handled, they're dangerous if your program needs to do any kind of safe shutdown or cleanup.

If you want your PHP code to install a signal handler, so you can control its exit status when signaled, see http://php.net/manual/en/function.pcntl-signal.php

answered Apr 18 '15 at 16:41

Charles Duffy

280,126
43
390
441

Thanks for spending some time on your response. As your response is so complete, I'm going to give you a little more information about my environment so you can give some more specific advice. I've come across everything you mentioned already and decided on using this method. So I would love to hear what you have to say. I run CentOS 6, please tell me of any other information you require. – youdsmedia Apr 18 '15 at 17:39
Yes I do. EPEL certainly. – youdsmedia Apr 19 '15 at 18:19
The supervisor package is available in EPEL. See http://supervisord.org/running.html#adding-a-program – Charles Duffy Apr 19 '15 at 18:26
Yeah but my question still stands, how to "respawn" a process? – youdsmedia Apr 19 '15 at 18:32
supervisord does that automatically. Anything you tell it to run, it starts on boot and restarts on death (unless configured otherwise). Read the fine manual to which you've been linked. – Charles Duffy Apr 19 '15 at 19:04
I'm not going to mark the question as the answer because it simply isn't, how to kill a process which is in an until loop and have the process respawn after receiving a suitable kill signal simply hasn't been answered, but I take your advice on board and will do as such. – youdsmedia Apr 20 '15 at 09:13
I disagree -- you haven't asked for clarification on those parts of the answer, but the text directly address them. There's a detailed explanation of why your existing until loop is unreliable and how to fix it (both by resolving bash syntax issues and ensuring that your PHP program exits with a nonzero status when signalled). – Charles Duffy Apr 20 '15 at 12:41

BASH: How monitor a script for execution failure

1 Answers1