1

I wrote bash script to calculate the hashes of a files, when it's file path given and queries one among the file hash to get the result of that hash from a API service ( Parsing using jq ) .

#Scanner.sh

#!/bin/bash

for i in $(find $1 -type f);do
    md5_checksum=$(md5sum $i|cut -d' ' -f 1)
    sha1_checksum=$(sha1sum $i|cut -d' ' -f 1)
    sha256_checksum=$(sha256sum $i|cut -d' ' -f 1)
    json_result="$(curl --silent -H 'Authorization:Basic XXXX' 'https://XXXXX.com/api/databrowser/malware_presence/query/sha1/'$sha1_checksum'?format=json&extended=true'| jq -r '.rl.malware_presence.name,.rl.malware_presence.level' | awk -vORS=, '{print $1}' |sed 's/,$/\n/')"
    echo "$md5_checksum,$sha1_checksum,$sha256_checksum,$json_result"
done


#Result : 
c63a1576e4b416e6944a1c39dbdfc3b4,fd55dfdd9a432e34ce6c593cd09a48f457f7aab6,e2d1c1975c2406f60f1c0fe5255560c2cd06434e669933536a86296a2eb70020,Malware,5

Now, it's taking too much time to process one and get results for one file hash ( 10 Sec ). How can i send 5 request per second and get the results faster ?

Any suggestions please ?

laurent
  • 88,262
  • 77
  • 290
  • 428
Arun
  • 1,160
  • 3
  • 17
  • 33

1 Answers1

0

You could put your code in a function and run it in the background, with something like this:

    runCurl() {
        md5_checksum=$(md5sum $1|cut -d' ' -f 1)
        sha1_checksum=$(sha1sum $1|cut -d' ' -f 1)
        sha256_checksum=$(sha256sum $1|cut -d' ' -f 1)
        json_result="$(curl --silent -H 'Authorization:Basic XXXX' 'https://XXXXX.com/api/databrowser/malware_presence/query/sha1/'$sha1_checksum'?format=json&extended=true'| jq -r '.rl.malware_presence.name,.rl.malware_presence.level' | awk -vORS=, '{print $1}' |sed 's/,$/\n/')"
        echo "$md5_checksum,$sha1_checksum,$sha256_checksum,$json_result"
    }

    for i in $(find $1 -type f);do
        runCurl $i &
    done
laurent
  • 88,262
  • 77
  • 290
  • 428
  • Thank you, This works. But using "&" might launch multiple requests. That might bombard the API service server. How can i rate limit ? For example , 50 queries per minute ? – Arun Nov 25 '16 at 12:13
  • You can use the `wait` command to wait for the current background tasks to finish. So maybe you could use this command every 5 requests. – laurent Nov 25 '16 at 12:24