Using `until` and `/usr/bin/timeout` in a script

Question

I want to perform a command that takes about 1 minute to finish, in a bash script. However, sometimes this command hangs, so I want to use /usr/bin/timeout inside a loop until it works.

If I use timeout 300 mycommand myarg1, it works, but if I use it inside bash in this loop below, it doesn't print anything (not even the typical output that my command prints) and it hangs!:

until timeout 300 mycommand myarg
do
    echo "The command timed out, trying again..."
done

My version of bash:

$ bash --version
GNU bash, version 4.3.48(1)-release (x86_64-pc-linux-gnu)
Copyright (C) 2013 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>

My version of timeout:

$ timeout --version
timeout (GNU coreutils) 8.25
Copyright (C) 2016 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>.
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.

(Standard Ubuntu16.04)

There is a mismatch between the argument value in the standalone invocation (`timeout 300 mycommand myarg1`) and the loop version (`until timeout 300 mycommand myarg`). Is it just a typo or you indeed have tried the two scenarios with different arguments? — Leon, Dec 21 '17 at 14:20
When I try it with a command that's just a bash script that sleeps 2 seconds and echoes some text, it works fine: `until timeout 1 cmd.sh; do ... ; done` repeats indefinitely, and `...timeout 3...` completes normally. You should try this, too. If it works in your environment, then the problem is likely to be the way your command handles `SIGTERM` and the terminal. One thing to try is the `-k` option of timeout. — Gene, Dec 27 '17 at 03:35

Ljm Dullaart · Answer 1 · 2017-12-22T16:24:12.620

The problem with

(mycommand args) & pid=$!
sleep 1000 && kill -INT $pid

is that it always takes 1000 seconds. Taking a more active (and CPU consuming) approach will shorten that time:

typeset -i i 
i=0
command with arguments &
pid=$!
while ps $pid > /dev/null 2>/dev/null ; do
    i=$i+1
    if [ $i -gt 999 ] ; then
        kill -9 $pid
    fi
    sleep 1
done

Or, if your system is not too busy or the interval is short:

command with arguments &
pid=$! 
echo "kill -9 $pid" | at "now + 15 minutes"
wait

But timeout will also work, of course.

The reason why

until timeout 300 mycommand myarg
do
    echo "The command timed out, trying again..."
done

hangs is that bash will try to evaluate the condition for your until, which may take upto 300 seconds. If timeout 300 mycommand myarg returns a success, the until is never executed. Try for example:

if timeout 30 mycommand myarg ; then
    echo "The timeout of mycommand succeeded"
else
    echo "It failed! What a pitty"
fi

score 2 · Answer 2 · answered Dec 28 '17 at 07:11

This works like a charm.

When command finish:

sh timeoutLoop.sh 4

OUTPUT:

Seconds start now...
3 seconds passed...

When command not finish:

sh timeoutLoop.sh 2

OUTPUT:

Seconds start now...
The command timed out, trying again...
Seconds start now...
The command timed out, trying again...
Seconds start now...
The command timed out, trying again...
Seconds start now...
The command timed out, trying again...
...

Your mycommand = waitSeconds.sh

#!/bin/bash -

echo "Seconds start now..."
sleep $1
echo "$1 seconds passed..."

And the loop bash timeoutLoop.sh

#!/bin/bash -

until timeout $1 waitSeconds.sh 3
do
echo 'The command timed out, trying again...'
done

Try this, and don't forget the header of the script.

sorak · Answer 3 · 2017-12-29T04:46:07.453

With this script you can configure the desired timeout as a variable. It launches the process in the background, grabs the process ID, and checks every second to see if the process is still running. If so, a dot is printed, so the process status is very visible. It will report completion of the background thread, and terminate it if it runs longer than desired. The loop in this configuration uses very little overhead.

timeout=300
wait() {
    sleep 1 # sleep for one second
    echo -n . # show that we are still waiting
    [ `ps|grep $pid|wc -l` -gt 0 ] && { # background process still running?
        [ `ps -p $pid -o etimes|grep [0-9]` -gt $timeout ] && { # running too long?
            echo "Time expired.";
            kill $pid;
        } || {
            wait; # keep waiting
        }
    } || {
        echo "DONE!";
    }
}
sleep 20 & # replace this with mycommand &
pid=$!
wait

this might work but it's really hacky and unreadable, so I won't give the bounty to this one sorry — knocte, Dec 29 '17 at 02:39

score 1 · Answer 4 · answered Dec 24 '17 at 09:50

I can't explain exactly what happen but it seem that when the time expire, timeout use kill which return 0 (success) so at done the shell look at $? and restart the process.

Your echo "The command timed out, trying again..." is exactly what happen.

So try this way.

until timeout 20 find / -name "*.sh" 2>/dev/null ;do
  echo "The command timed out, not trying again..."
  break
done

WLGfx · Answer 5 · 2017-12-29T10:00:34.447

0

I recently had to find a solution to killing a process which wouldn't finish. This is what I managed to come up with:

(mycommand args) & pid=$!
sleep 1000 && kill -INT $pid

Should be self explanatory. Runs your command and grabs the process ID. Kills it after the timeout.

My specific use was for scanning DVB-S:

(w_scan -a 7 -E 0 -f s -c $1 -s $2 -o 7 -x >$tunefile) & pid=$!
sleep 1500 && kill -INT $pid

The w_scan command would never finish and would loop forever scanning the same frequencies.

EDIT: You can check if the pid is still running: This will at least allow your script to act accordingly.

if ps -p $id > /dev/null
then 
    : active
else
    : inactive
fi

EDIT2: I just ran a quick script using ffmpeg to convert from one format to another. My test converted an mkv (2Gb file) to mp4. This would normally take a very long time but I wanted to get it working with just 10 seconds and then exits it.

It wasn't much, but the initial test ran fine.

film=/home/wlgfx/longfilm.mkv
output=/home/wlgfx/longfilm.mp4

(ffmpeg -i $film -y -vcodec copy -acodec copy $output) & pid=$!
#for i in 'seq 1 10' # 10 seconds
#do
#    sleep 1
    if wait $pid; then echo "$pid success $?"
    else echo "$pid fail $?"
    fi
#done

Exit status is $?

I also noticed that ffmpeg only took about 5 seconds to convert the 2Gb file into another container. I'll make an update so that it transcodes and I'll make further changes to the script to reflect them so that it will kill the process after x seconds.

Currently, I'm looking here: https://stackoverflow.com/a/1570351/2979092

EDIT4: By running a separate timeout process, this time, if your process exits properly within the timeout, then the successful exit code will be displayed. Otherwise it will terminate the hanging process.

film=/home/wlgfx/longfilm.mkv
output=/home/wlgfx/longfilm.mp4

while : ; do

    #(ffmpeg -i $film -y -vcodec mpeg2video -acodec copy $output) & pid=$!
    (ffmpeg -i $film -y -vcodec copy -acodec copy $output) & pid=$!

    (sleep 25 ; echo 'timeout'; kill $pid) & killpid=$!

    wait $pid # will get status if completed successfully
    status=$?

    (kill -0 $killpid && kill $killpid) || true

    [[ $status == 0 ]] || break

done

edited Dec 29 '17 at 10:00

answered Dec 21 '17 at 14:49

WLGfx

1,169
15
31

I'm sorry but this only would cover the case of a command that always hangs; in my answer I said that it sometimes hangs, but not always – knocte Dec 22 '17 at 09:49
Just added the conditional check on the pid. – WLGfx Dec 22 '17 at 10:14
but what if the pid number was taken by another process in the machine? wouldn't it be better to check the exit code of the process? – knocte Dec 26 '17 at 10:29
(I wasn't the one downvoting btw) – knocte Dec 27 '17 at 09:16
Unless you have a lot of processes being spawned, it's not likely the PID will change to another process in such a short time. I'll review the code today and update it. – WLGfx Dec 27 '17 at 10:24
bounty expires in one day, if you can check the exit code of the finished pid instead of trying to kill it no matter what, I'll give you the bounty – knocte Dec 27 '17 at 16:17
I've added a quick ffmpeg 10 second run. After dinner I'll add the exit code. Chaos here. – WLGfx Dec 27 '17 at 16:58
your EDIT3 doesn't try again if it fails at the specified timeout – knocte Dec 28 '17 at 10:30
Taken the example from https://stackoverflow.com/a/22384727/2979092 I finally got it working properly. – WLGfx Dec 28 '17 at 11:11
but it's not trying again, dude, it needs a loop :) – knocte Dec 29 '17 at 02:32
Now it will keep running until it gets an exit status of 0 – WLGfx Dec 29 '17 at 10:01
this is, again, killing the process without knowing if it finished properly – knocte Dec 29 '17 at 12:31

Using `until` and `/usr/bin/timeout` in a script

5 Answers5