0

txt with more than 30000 records. All records are one for line and is an IP like this:

192.168.0.1
192.168.0.2
192.168.0.3
192.168.0.4
192.168.0.5
192.168.0.6
192.168.0.7
192.168.0.8
192.168.0.9
192.168.0.10

I read each row in a bash script, and I need to run a curl like this:

while IFS= read -r line || [[ -n "$line" ]]; do
    #check_site "$line"
    resp=$(curl -i -m1 http://$line 2>&1)
    echo "$resp" | grep -Eo "$ok" > /dev/null

    if [ $? -ne 0 ]; then
        #echo -e "failed: $line" >> "${logfile}"
        echo -e "Command: curl -i -m1 http://$line 2>&1" >> "${outfile}"
        echo -e "failed: $line:\n\n \"$resp\"\n\n" >> "${outfile}"
        echo "$line" >> "${faillog}"
    fi
done < "${FILE}"

Is there a method to run multiple lines simultaneously in my file to reduce the execution time?

Daniel Widdis
  • 8,424
  • 13
  • 41
  • 63
Marco Ferrara
  • 516
  • 1
  • 8
  • 26
  • 2
    Sure, check tag for **GNU Parallel** by inputting `[gnu-parallel]` in search box. – Mark Setchell Nov 11 '21 at 09:52
  • 1
    Check your script with shellcheck. `if [ $? -ne 0 ]; then` do `if grep -q <<<"$resp" ... ; then`. – KamilCuk Nov 11 '21 at 10:00
  • Hi KamilCuk, thanks for your reply. I'm sorry but I don't understand your advise. could I have an example with my script, please? – Marco Ferrara Nov 11 '21 at 10:26
  • The `[ $? -ne 0 ]` is not testing the return code of `curl` but the one of `grep` – Fravadona Nov 11 '21 at 11:42
  • Yes it's correct, I need to check the return value of the `grep` (only acceptable response in `$ok` variable, if not write to log file). – Marco Ferrara Nov 11 '21 at 12:20
  • Multiprocessing in Bash can be found [in many different flavors](https://stackoverflow.com/a/69452918/8584929) on SO. Why not just tweak one of the existing solutions? – Andrej Podzimek Nov 12 '21 at 04:53

3 Answers3

3

I solved for the multiprocess in this way:

#export variable to be used into function    
export outlog="/tmp/out.log"
export faillog="/tmp/fail.log"
export ok="(curl: \(7\) Failed to connect to)" # acceptable responses

# create function:
check_site() {
  ip=$1
  resp=$(curl -i -m1 http://$ip 2>&1)
  echo "$resp" | grep -Eo "$ok" > /dev/null

  if [ $? -ne 0 ]; then
    echo -e "Command: curl -i -m1 http://$ip 2>&1" >> "${outlog}"
    echo -e "Block failed: $ip:\n\n \"$resp\"\n\n" >> "${outlog}"
    echo "$ip" >> "${faillog}"
  fi
}

# call the function:
export -f check_site
parallel -j 252 -a "${FILE}" check_site
Marco Ferrara
  • 516
  • 1
  • 8
  • 26
1

Xargs will do the trick. Wikipedia

jwan
  • 135
  • 7
-1

This article describe approach to resolve parallel execution, it may help you: Parallel execution in Bash

Example from the article:

#!/bin/bash

RANDOM=10
JOBS_COUNTER=0
MAX_CHILDREN=10
MY_PID=$$

for i in {1..100}
do
    echo Cycle counter: $i
    JOBS_COUNTER=$((`ps ax -Ao ppid | grep $MY_PID | wc -l`))
    while [ $JOBS_COUNTER -ge $MAX_CHILDREN ]
    do
        JOBS_COUNTER=$((`ps ax -Ao ppid | grep $MY_PID | wc -l`))
        echo Jobs counter: $JOBS_COUNTER
        sleep 1
    done
    sleep $(($RANDOM % 30)) &
done
echo Finishing children ...
# wait for children here
while [ $JOBS_COUNTER -gt 1 ]
do
    JOBS_COUNTER=$((`ps ax -Ao ppid | grep $MY_PID | wc -l`))
    echo Jobs counter: $JOBS_COUNTER
    sleep 1
done
echo Done
Podrepny
  • 29
  • 3