how to submit multiple jobs using a single .sh file

Question

I am using WinSCP SSH Client to connect my Windows PC to a HPC system. I created a file called RUN1.sh including following code:

#PBS -S /bin/bash                        
#PBS -q batch
#PBS -N Rjobname
#PBS -l nodes=1:ppn=1:AMD
#PBS -l walltime=400:00:00
#PBS -l mem=20gb

cd $PBS_O_WORKDIR

module load mplus/7.4

mplus -i mplus1.inp -o mplus1.out > output_${PBS_JOBID}.log

It simply calls mplus1.inp file and save results of the analysis in the mplus1.out file. I can do it on Linux one by one. I can do the same for mplus2.inp file running RUN2.sh file below:

#PBS -S /bin/bash                        
#PBS -q batch
#PBS -N Rjobname
#PBS -l nodes=1:ppn=1:AMD
#PBS -l walltime=400:00:00
#PBS -l mem=20gb

cd $PBS_O_WORKDIR

module load mplus/7.4

mplus -i mplus2.inp -o mplus2.out > output_${PBS_JOBID}.log

However, I have 400 files like this (RUN1.sh, RUN2.sh, ......RUN400.sh). I was wondering if there is a way to create a single file in order to run all 400 jobs in linux.

You are looking for a loop. There's many answer that talk about how to use loops. Such as https://stackoverflow.com/questions/169511/. You want to wrap the `mplus...` line into a loop. — omajid, May 08 '20 at 00:16

score 0 · Answer 1 · answered Aug 04 '23 at 09:09

This is the typical use case for a job array. Modify your submission script as :

#PBS -S /bin/bash                        
#PBS -q batch
#PBS -N Rjobname
#PBS -l nodes=1:ppn=1:AMD
#PBS -l walltime=400:00:00
#PBS -l mem=20gb
#PBS -J 1-400

cd $PBS_O_WORKDIR

module load mplus/7.4

mplus -i mplus${PBS_ARRAY_INDEX}.inp -o mplus${PBS_ARRAY_INDEX}.out > output_${PBS_JOBID}.log

The -J line will create 400 jobs, and the ${PBS_ARRAY_INDEX} variable will have a distinct value between 1 and 400 in each of them.

how to submit multiple jobs using a single .sh file

1 Answers1