49

how can I parametrize a shell script that is executed on a grid (started with qsub) ? I have a shell script, where I use getopts to read the parameters.

When I start (qsub script.sh -r firstparam -s secondparam ..) this working script with qsub I receive error messages,

qsub: invalid option -- s

qsub: illegal -r value

as qsub thinks the parameter are for itself. Yet I have not found any solution.

Thanks

Charles
  • 50,943
  • 13
  • 104
  • 142
Martin
  • 751
  • 1
  • 5
  • 8
  • 1
    I just figured out how to solve it: just print the commands of the shell scrip with echo and pipe the result to qsub: ./script.sh | qsub – Martin Aug 17 '10 at 15:34
  • You should post your solution as an answer. Then come back and mark it accepted. By the way, did you try quoting the argument to `qsub`? `qsub 'script.sh -r firstparam -s secondparam'` I have no idea if that works in this case. – Dennis Williamson Aug 17 '10 at 19:06
  • I've seen more than one command named `qsub`; are you referring to [this one](http://www.clusterresources.com/torquedocs/commands/qsub.shtml)? – Keith Thompson Jan 16 '12 at 06:10

4 Answers4

45

Using the qsub -v option is the proper way:

qsub -v par_name=par_value[,par_name=par_value...] script.sh

par_name can be used as variable in the shell script.

volks
  • 451
  • 1
  • 4
  • 2
  • Thanks for your suggestion! In my opinion the problem with this approach is the difference of the parameter handling for the use of one application as standalone or distributed app. Therefore I prefer the style described above. – Martin Jan 21 '11 at 10:41
  • 15
    What if I don't care about the names of variables (because in my script I use just $1, $2 and so on)? To be more precise, all I want is just to pass arguments to the script. – Perlnika Feb 25 '13 at 10:52
  • 3
    I also don't care about the names of the variables - I prefer the script to parse the options as it normally would when running directly from a terminal, even when run with qsub. – David Doria Apr 18 '14 at 16:55
38

In addition to volk's answer, in order to reference the variables in the list (designated by -v) you simply use the name you define in your call. So, say you made a call to qsub as follows

qsub -v foo='qux' myRunScript.sh

Then myRunScript.sh could look something like this: 

#!/bin/bash
#PBS -l nodes=1:ppn=16,walltime=0:00:59
#PBS -l mem=62000mb
#PBS -m abe

bar=${foo}
echo "${bar}"

Where the output would be 

qux

Hope this helps!

Charles Duffy
  • 280,126
  • 43
  • 390
  • 441
jhrf
  • 980
  • 9
  • 17
  • 1
    @_jhrf I think you wanted `$bar` – Manolete Aug 18 '14 at 10:17
  • You're right. I've looked at this answer many times and not seen that error! Thanks. – jhrf Aug 26 '14 at 12:59
  • 3
    @Manolete and @jhrf because you specifically gave `bash`, it does not matter in this case. Though the ` = ` (with spaces **is** a syntax problem...). See [this excellent answer](https://stackoverflow.com/a/18136920/3814202) for why `$bar` and `${bar}` both work here :) – svenevs Aug 20 '17 at 18:03
22

I just figured out how to solve it: just print the commands of the shell scrip with echo and pipe the result to qsub:

echo "./script.sh var1=13 var2=24" | qsub

Martin
  • 751
  • 1
  • 5
  • 8
  • your answer needs to be more explicit - come up with an example – Jeremy Leipzig Nov 23 '10 at 15:41
  • It seems my [problem](http://stackoverflow.com/questions/11354186/using-a-loop-variable-in-a-bash-script-to-pass-different-command-line-args/11354348#11354348), is similar. How do I need to change my bash script? Can you give an example. – lovespeed Jul 06 '12 at 00:52
  • The problem in this answer is that the job name is always STDIN (as appears in qstat -a), at least for my case – hammady Jan 14 '14 at 08:42
  • 1
    @hammady just use `echo "./script.sh var1=13 var2=24" | qsub -N myjobname` – Mmmh mmh Aug 20 '14 at 11:14
  • I'm afraid this just runs `script.sh` on the node where `qsub` is executed, not on the cluster. – a06e Jan 28 '15 at 19:01
  • In my qsub version (probably Univa Grid Engine, this is written in man, but no --version option exists) - there is nothing written about reading input. I'm afraid that's not portable and reliable. Anyway, glad that it works for you. – Yaroslav Nikitenko Mar 29 '20 at 18:52
20

There is a better way...

I'm really surprised at how long this question has gone without a good answer. It may be that the specific version of qsub wasn't specified. qsub exists in at least Torque and also Sun Grid Engine, maybe other schedulers. So, it's important to know which you're using. I'll talk about a few here:

TORQUE: qsub -F <arguments> command

man page
Here's an example of how I normally use it. Starting with this example script which just echoes any arguments passed to it:

$ cat testArgs.pbs
#!/usr/bin/env bash

echo $@

I would submit the job like this: 

$ qsub -F "--here are the --args" testArgs.pbs
3883919.pnap-mgt1.cm.cluster

And this is what the output file looks like after it runs:

$ cat testArgs.pbs.o3883919
--here are the --args

Sun Grid Engine: qsub command [ command_args ]

man page
You just add the arguments after the command, same as you would when executing in the shell. I don't have SGE running anywhere, so no example for this one. But it's the same with Slurm, which is below

Slurm: sbatch command [ command_args ]

man page
Here I submit the same script I used with the Torque example above:

$ sbatch testArgs.sh what the heck
Submitted batch job 104331

And the results: 

$ cat slurm-104331.out
what the heck

Exporting environment variables != passing arguments

Exporting environment variables is very different from passing arguments to a command.
Here is a good discussion on the differences.

The qsub answers above all recommend -v. To be clear, -v exports environment variables, -F passes arguments to the command.

I generally prefer to parameterize my scripts by allowing for arguments. In fact, I would say it's much more common to use scripts like this process_data.sh --threads 8 than doing something like export THREADS=8; process_data.sh.

calico_
  • 1,171
  • 12
  • 23
  • 1
    Thanks! In my environment, man qsub says there is no -F option, but I tried anyway and it works! Probably this option was added recently. But it is the right answer to this question. – Fil Jan 26 '18 at 04:02
  • thanks! I am using qsub (Sun Grid Engine) and it works! note that you don't need to add "command" as a key word. ```qsub script.sge arguments``` – Pengyao Feb 22 '18 at 02:04