1

I'm new to bash scripting. I wrote a script that runs 4 Python scripts in parallel. These scripts make API calls to bring data to the local machine, sometimes these scripts fail, and get only a fraction of the data.

How can I make sure that if a script fails, execution of bash script will start over until the script get all the data?

#!/bin/bash

scripts=/root/scripts
data=/root/data
date=$(date +"%Y_%m_%d")

$scripts/filesystem.py > $data/filesystem_$date.log &
$scripts/cpu.py > $data/cpu_$date.log &
$scripts/memoria.py > $data/memoria_$date.log &
$scripts/swap.py > $data/swap_$date.log &

wait

UPDATE

based on your comments the script will be something like this? but the parallel execution will still work? the wait command at the end is necessary?

until [ "$?" -eq "0" ]
do
    $scripts/filesystem.py > $datos/filesystem_$fecha.log &
done

until [ "$?" -eq "0" ]
do
    $scripts/cpu.py > $datos/cpu_$fecha.log &
done

until [ "$?" -eq "0" ]
do
    $scripts/memoria.py > $datos/memoria_$fecha.log &
done

until [ "$?" -eq "0" ]
do
    $scripts/swap.py > $datos/swap_$fecha.log &

done
wait
Mefistofeles
  • 79
  • 6
  • 2
    Need to understand a couple of things: 1.By convention, scripts should exit with a `0` status on success and some other non-zero status on error. 2. background process pid can be obtained from `$!`. 3. `wait` can be used to wait for a process to exit and then get its exit status from `$?`. Combine all that into a loop which waits for each process, checks its exit status and then re-runs the process if exit status indicates failure. – kaylum Dec 31 '19 at 05:29
  • 1
    If your individual scripts are well-behaved, they will exit with a non-zero exit status when something goes wrong. Then the Bash part will be easy. But do they currently do that? – tripleee Dec 31 '19 at 06:57
  • yes the scripts work well, but sometimes, the connection with the remote machine is interrupted, and thats when I get the errors – Mefistofeles Dec 31 '19 at 18:02

2 Answers2

1

Use function with recursion

f1 () { $scripts/filesystem.py > $data/filesystem_$date.log || f1; }
f1 &

And so on for all scripts If scrip fails it'll start over again

Ivan
  • 6,188
  • 1
  • 16
  • 23
  • I won't downvote a well-intentioned beginner answer but this really is rather convoluted. A regular `while` loop would be more idiomatic. – tripleee Dec 31 '19 at 06:59
-1

In above code you calling Python program.

https://superuser.com/questions/442745/do-commands-in-a-bash-script-run-in-parallel-or-one-after-another

https://unix.stackexchange.com/questions/242087/run-commands-in-parallel-and-wait-for-one-group-of-commands-to-finish-before-sta

#!/bin/bash

#Program 1

sleep 5 &
echo $?

if [ $? -ne 0 ]
then
exit 1
fi

#Program 2
sleep 3 &
echo $?

if [ $? = 0 ]
then
echo "command success"
else
exit 1
fi


wait
echo "Hello World "

In case Program 1 or Program 2 fail, have to write Function call, which will retry the operation.

Question is to identify the cause of failure of program 1 or 2, prior rerun

tripleee
  • 175,061
  • 34
  • 275
  • 318
Vishal
  • 44
  • 7
  • 1
    The `$?` in the `if` will be the exit code from `echo`. The `$?` before that will not reflect the exit status from the background process because it is not available when you run in the background. This does not appear to attempt to answer the question at the top of this page anyway. – tripleee Dec 31 '19 at 07:14
  • Thank's for feedback .Will try to understand the question in near future, based on that will attempt to answer.In this scenario 3 script provoked parallell , in case of failure reported -reattempt has to performed .Was in impression logic has to be encapsulated based on cause of failure , to retry or to fix that first – Vishal Dec 31 '19 at 07:23
  • 1
    Explicitly comparing `$?` is an antipattern anyway; the shell's flow control statements do this exactly but implicitly. See also https://stackoverflow.com/questions/36313216/why-is-testing-to-see-if-a-command-succeeded-or-not-an-anti-pattern – tripleee Dec 31 '19 at 07:39