0
#!/usr/bin/env bash
exec 3>&1

fun_1(){

    urlcount=$(wc -l < list.txt)
    loopcount=0
    for url in $(cat list.txt);
        do
        ((loopcount++))
        echo -e "\nProcessing URL #${loopcount} (of ${urlcount}) [ ${url} ]\n"

        #the below curl command is the problem which i need to time it to maximum 5 minuts the continue the loop (because sometime it could take massive time to complete)
        curl -s "http://localhost:5555/?url=$url"

        # check for api status percentage
            until [[ $(curl -s "http://localhost:5555/view/status" |  jq -r '.status') == "100" ]]
            do
            echo -e "\n[-] Waiting for command $url\n"
            sleep 5 || break
            done
        curl -s "http://localhost:5555/results" | jq -r '.results[]' >> results.txt
    
    done

}

for domain in "$@"
    do
    fun_1 $domain 2>&1 >&3 | tee -a $WORKING_DIR/error_log.txt
done

This script has multiple functions like fun_1 which is execute one after another. The problem is some functions which have a loop function using for loop or while loop could be running for very long time, which is exhausting my server (VPS) and of course waste of time.

THE QUESTION is can I time this function to run for a certain time like one or two hour as maximum?

tripleee
  • 175,061
  • 34
  • 275
  • 318
Emad
  • 143
  • 8

1 Answers1

1

You could check how many seconds the script is running since start time, using the $SECONDS variable, and see if it's greater than some defined deadline.

I have replicated your execution flow while swapping something for dummy functionality.

#!/usr/bin/env bash
exec 3>&1

fun_1() {

  domain="$1"
  deadline="$2"

  for i in {1..100}; do
    echo "iteration $i"

    # try until deadline is exceeded or curl succeeds
    curl_success=0
    until [ "$SECONDS" -gt "$deadline" ] || [ "$curl_success" -eq 1 ]; do
      echo "retrying..."
      sleep 5
    done

    # if deadline is exceeded, break out of the loop
    if [ "$SECONDS" -gt "$deadline" ]; then
      echo "Deadline exeeded"
      break
    fi

    # curl results if deadline not exeeded
    echo "curling results..."

  done
}


deadline=$((SECONDS + 10))

for domain in "$@"; do
  # if deadline is exceeded, break out of the loop
  if [ "$SECONDS" -gt "$deadline" ]; then
    echo "Deadline exeeded"
    break
  fi
  fun_1 "$domain" "$deadline" 2>&1 >&3 | tee -a ./error_log.txt
done

You could also try to do some tricks with timeout. For example, if you can't modify your function to build a timeout into it.

The Fool
  • 16,715
  • 5
  • 52
  • 86
  • thx for helping i have edit the post with function which i have problem with , it contains two loops inside others one for check api status and the main one for api call url – Emad Jan 29 '22 at 14:45
  • @Emad I have adjusted my answer accordingly. You have a loop in a loop in a loop. Would be nice to have context cancellation like in golang. – The Fool Jan 29 '22 at 15:41
  • 1
    Note that you can get time since a bash interpreter's startup by checking `$SECONDS` much more efficiently than invoking the external `date` command. (Also, bash 4.3 and newer support `printf -v current_time '%(%s)T\n' -1`, putting the epoch time in `"$current_time"` with no `fork()` operations at all). – Charles Duffy Jan 29 '22 at 15:44
  • @CharlesDuffy, amazing tip. Thank you. This will come in handy in the future. I have updated my answer accordingly. – The Fool Jan 29 '22 at 16:22
  • i see a lot of if statement and loops, where exactly my command will pace in this code ? – Emad Jan 29 '22 at 18:38
  • This is your whole code @Emad. I just removed the curl calls and stuff because I obviously can't use curl, and I don't have a text file with server and so on. But the general algorithm is still your code. If you watch closely, you should be able to recognize that its structure is identical to yours. – The Fool Jan 29 '22 at 18:53
  • There are also just 2 if call. and an additional condition in the until loop. – The Fool Jan 29 '22 at 19:12