2

I want to use GNU parallel to execute some commands in parallel, but I don't know how to send the argument to my bash scripts.

My bash_script.sh:

scp $1.zip xxx@1.com:~/
scp $1.zip xxx@2.com:~/
scp $1.zip xxx@3.com:~/
scp $1.zip xxx@4.com:~/
scp $1.zip xxx@5.com:~/

I can send the argument to the bash_script and execute it in sequence.

bash bash_script.sh argument

but how to do it in parallel?

parallel -a bash_script argument
Sunchi
  • 85
  • 1
  • 9
  • I hope the bash_script is not a static text, and I want to use an argument outside the script. – Sunchi Oct 29 '20 at 18:48

4 Answers4

1

parallel can execute individual processes in parallel, but a script is usually intended to be run as a single process.

You could do

parallel scp "$1" xxx@{}1.xyz.com: ::: {1..5}

or if you really wanted to split the script file into individual lines

sed "s#\$1#$1#g" bash_script.sh | parallel

If the script was just static text, you could simply have parallel read it line by line; but parallel doesn't have a value for $1 in that context, so we substitute it in, and then piped the substituted lines for parallel execution.

Tangentially, notice proper variable quoting and don't put a .sh extension on a Bash script. (Don't put any extension on any script, basically.)

tripleee
  • 175,061
  • 34
  • 275
  • 318
0

Maybe something like this:

for server in server1 server2 server 3
do
   scp "$1" $server: &
done

The & causes the job to run in the background. If I'm not understanding your requirements, please clarify.

ShawnMilo
  • 5,896
  • 3
  • 19
  • 15
0

The solution that worked for me is writing the commands to a text file (e.g: commands.txt) and the executing parallel:

parallel -j 2 < commands.txt
Gonzalo Matheu
  • 8,984
  • 5
  • 35
  • 58
0

bash_script.sh

parallel scp "$1" xxx@{}.com: ::: {1..5}

Usage:

bash bash_script.sh argument
Sunchi
  • 85
  • 1
  • 9
  • This basically duplicates my answer (except you had botched the quoting). – tripleee Oct 29 '20 at 19:35
  • (The `~/` is redundant because `scp` by default copies to the user's home directory.) – tripleee Oct 29 '20 at 19:50
  • Your answer just give me the idea that I should put the parallel command inside the script rather than execute in shell. The concept is different. Your answer doesn't set the argument as a dynamic script, and I need a dynamic one and that's the whole point. – Sunchi Oct 29 '20 at 20:17
  • Please review the link I provided which explains why removing the quotes is wrong. Your script will not work with file names which contain whitespace, wildcard characters, or other shell metacharacters. – tripleee Oct 29 '20 at 21:02
  • Anything you can type at the prompt, you can put in a script, and vice versa. – tripleee Oct 29 '20 at 21:02