How to use variables on qsub?

Question

I'm new to qsub and I'm trying to figure out how to use the task queue optimally. I have this script that works well:

#!bin/bash

##PBS -V  # Export all environment variables from the qsub command environment to the batch job.
#PBS -N run
#PBS -q normal.q 
#PBS -e archivo.err
#PBS -o archivo.out
#PBS -pe mpirun 8
#PBS -d ~/ # Working directory (PBS_O_WORKDIR)
#PBS -l nodes=1:ppn=8    

~/brinicle/step-2/onephase_3/./main.x --mesh ~/brinicle/step-2/onephase_3/results/mesh.msh -Rmin 0 -Rmax 10 -Zmin 0 -Zmax 10 -o 2 -r 2 -T_f -10 -a_l 7.8 -a_s 70.8 -dt 0.01 -t_f 1 -v_s 10 -ode 12 -reltol 0.00001 -abstol 0.00001

The problem, as you can see, is that the command line is huge and hard to edit from the command shell. I would want to separate it into variables such as

#MESH="--mesh ~/brinicle/step-2/onephase_3/results/mesh.msh"
#EXE="~/brinicle/step-2/onephase_3/./main.x"
.
.
.
$EXE $MESH $PARAMETERS

And for the other parameters too.

But when I do this the program doesn't run and says that there's an illegal variable or that the variable is undefined. Also, is very important to me to change easily the parameters -o, -r, -ode and send multiple jobs at once. For example 5 equal jobs with -o 1 then 5 with -0 2 and so on. I want to be also able to modify in this way -r and -ode. The problem is that without using the variables I really don't know how to do that.

Please, if someone can tell me how to automate the script in this way would be a huge help.

https://stackoverflow.com/questions/5615717/how-to-store-a-command-in-a-variable-in-a-shell-script https://stackoverflow.com/questions/4651437/how-do-i-set-a-variable-to-the-output-of-a-command-in-bash https://stackoverflow.com/questions/2005192/how-to-execute-a-bash-command-stored-as-a-string-with-quotes-and-asterisk https://stackoverflow.com/questions/2355148/run-a-string-as-a-command-within-a-bash-script — KamilCuk, May 06 '21 at 06:40

score 0 · Answer 1 · answered May 06 '21 at 06:42

0

Use bash arrays.

exe=(~/brinicle/step-2/onephase_3/./main.x)
mesh=(--mesh ~/brinicle/step-2/onephase_3/results/mesh.msh)
parms=(
    -Rmin 0
    -Rmax 10
    -Zmin 0
    -Zmax 10
    . etc.
)
"${exe[@]}" "${mesh[@]}" "${parms[@]}"

Research bash arrays and how to use then and quoting in shell. Prefer to use lower case variables. Research order of expansions in shell.

answered May 06 '21 at 06:42

KamilCuk

120,984
8
59
111

I tied this and it says this: Badly placed ()'s. – Carlos Andrés del Valle May 06 '21 at 12:24
`one after the other in the same node?` no idea, I never used qsub. `this: Badly placed ()'s` Most probably you are executing the script under sh, not bash. The presented code [works here](https://replit.com/@kamilcukrowski/TraumaticSecondhandGravity#main.sh) – KamilCuk May 06 '21 at 17:20
1

`sh` also does not produce that message; it comes from `csh` or `tcsh` – tripleee May 07 '21 at 04:25

score 0 · Answer 2 · answered May 06 '21 at 07:14

0

One alternative if you have a lot of static parameters and a lot of dynamic ones is to refactor into a function where you hard-code what doesn't change, and interpolate the parts which do change.

qrunmesh () {
    qsub <<:
#!bin/bash

##PBS -V  # Export all environment variables from the qsub command environment to the batch job.
#PBS -N run
#PBS -q normal.q 
#PBS -e archivo.err
#PBS -o archivo.out
#PBS -pe mpirun 8
#PBS -d ~/ # Working directory (PBS_O_WORKDIR)
#PBS -l nodes=1:ppn=8

"$1" --mesh "$2" -Rmin 0 -Rmax 10 -Zmin 0 -Zmax 10 \
     -o "$3" -r "$4" -T_f -10 -a_l 7.8 -a_s 70.8 \
     -dt 0.01 -t_f 1 -v_s 10 -ode "$5" \
     -reltol 0.00001 -abstol 0.00001
:
}

for o in 1 2 3; do
  for r in 5 10 15; do
    for x in onephase_3 onephase_2 twophase_3; do
      for ode in 12 13 15; do
        for mesh in onephase_3 otherphase_2; do
           qrunmesh "$x" "$mesh" "$o" "$r" "$ode"
        done
      done
    done
  done
done

(I'm not very familiar with qsub; I assume it accepts the script on standard input if you don't pass in a script name. If not, maybe you have to store the here document in a temporary file, submit it, and remove the temporary file.)

answered May 06 '21 at 07:14

tripleee

175,061
34
275
318

Thanks a lot for your answer. It helps me a lot. Do you know if this will create a queue of tasks one after the other in the same node? Or the tasks will distribute evenly between all nodes? – Carlos Andrés del Valle May 06 '21 at 12:04
I have no idea about that; I would expect it to run on different nodes but I guess that also depends on the queue configuration etc. It's been many years since I used `qsub`. – tripleee May 06 '21 at 12:06
I tied this and it says this: Badly placed ()'s. – Carlos Andrés del Valle May 06 '21 at 12:24
Tried it where? This function definition syntax should be portable way back to early Bourne shell. – tripleee May 06 '21 at 12:27
The error message seems to come from `tcsh` which is incompatible with both the shells you have tagged in your question. – tripleee May 06 '21 at 12:32
I found the mistake, apparently I had to tell qsub what type of script I was using with the line # -S /bin/bash and it worked. Thanks – Carlos Andrés del Valle May 06 '21 at 14:02
You would run this locally in your own shell anyway; if you send this to `qsub` you are asking it to schedule new jobs for itself. – tripleee May 06 '21 at 14:12

How to use variables on qsub?

2 Answers2