UNIX execute parallel curls base on script parameter

Question

I have to send some cURL request in a parallel way. At one side I have the curl.sh script (for security reasons data has been hidden):

#!/bin/bash

SERVER=...
PORT=...
CONTEXTPATH=...
SERVICE=...
CERTIFICATETYPE=...
CERTIFICATE=...
KEYCERT=...
HEADER=...
CONTENT="Content-Type:text/xml;charset=UTF-8"

if [ "$1" != "" ] && [ "$2" != "" ]
then
        echo "Executing curl script for $1 > $2"
        curl -s -d @$1 --cert $CERTIFICATE:$KEYCERT --cert-type $CERTIFICATETYPE http://$SERVER:$PORT/$CONTEXTPATH/$SERVICE -H $HEADER -H $CONTENT > $2
else
        echo "ERROR not valid arguments: curl.sh[ARG1=$1,ARG2=$2]"
fi

So this script, will be call as:

$ curl.sh requestData outputFile

Then, in another parallel.sh script I have:

#!/bin/bash
MODE=$1
REQUEST=

if [ "$MODE" != "" ]
then
        if [ $MODE -eq 1 ]
        then
                REQUEST=requestFileOne.xml
        fi
        if [ $MODE -eq 2 ]
        then
                REQUEST=requestFileTwo.xml
        fi
else
        echo "ERROR no valid parameter"
        echo "./parallel.sh 1|3"
        echo "  mode 1: send request 1x1"
        echo "  mode 3: send request 3x3"
fi

if [ "$REQUEST" != "" ]
then
        echo "START CURL PARALLEL for $REQUEST"

        time ./curl.sh $REQUEST ./out/out_1.xml & ./curl.sh $REQUEST ./out/out_2.xml &
        ./curl.sh $REQUEST ./out/out_3.xml & ./curl.sh $REQUEST ./out/out_4.xml &
        ./curl.sh $REQUEST ./out/out_5.xml & ./curl.sh $REQUEST ./out/out_6.xml

        wait
fi

echo "END CURL PARALLEL"

As you can see, now I do

$ ./parallel.sh 1

But if I want to change the number of parallel calls, I have to edit the parallel.sh script file, and I would want something like:

$ ./parallel.sh file number

Where {file: 1|2} is the data file to use with cURL, and {number:X} will be the number of parallel calls.

Firstly, I thought of building the parallel executing command inside a loop

for i in 1..$2
do
   stringVar="$stringVar | ./curl.sh $REQUEST ./out/output_$i.xml"
done

for i in 1..$2
do
   result=`./curl.sh $REQUEST ./out/output_$i.xml`
done

for i in 1..$2
do
   ./curl.sh $REQUEST ./out/output_$i.xml
done

But, obviously none of this loops works as expected. At first loop, I was trying to save the curls command as String, so later I will run the String as command.

Any idea? Thanks in advance.

What about using [getops](http://stackoverflow.com/a/11279500/1983854)? This way you can execute it like `./parallel.sh -f file -n number` and define default values. — fedorqui, Jun 11 '13 at 10:02
That would be nice If I know how many times the cURL script must be called. But user may could use for just a single call (-n 1) [non-parallel], single pair (-n 2), two pairs (-n 4) and so on. — Wolfchamane, Jun 11 '13 at 10:13

score 0 · Answer 1 · answered Jun 13 '13 at 20:41

0

Can you use GNU Parallel:

seq 100 | parallel -j0 curl.sh file.xml output{}.xml

Watch the intro videos for GNU Parallel to learn more: https://www.youtube.com/playlist?list=PL284C9FF2488BC6D1

10 seconds installation:

wget -O - pi.dk/3 | bash

answered Jun 13 '13 at 20:41

Ole Tange

31,768
5
86
104

UNIX execute parallel curls base on script parameter

1 Answers1