Is there a "one-liner" for submitting many jobs to SLURM (similar to LSF)?

Question

Can I submit "one-liners" to SLURM?

Using bsub from LSF and the standard Linux utility xargs, I can easily submit a separate job for uncompressing all of the files in a directory:

ls *.gz | sed 's/.gz$//g' | xargs -I {} bsub 'gunzip -c {}.gz > {}'

Using SLURM, I thought that srun or sbatch would be work, but to no avail:

ls *.gz | sed 's/.gz$//g' | xargs -I {}  srun 'gunzip -c {}.gz > {}'
gzip: srun: error: compute-node-01: task 0: Exited with exit code 1
stdin: unexpected end of file

ls *.gz | sed 's/.gz$//g' | xargs -I {}  sbatch 'gunzip -c {}.gz > {}'
sbatch: error: Unable to open file gunzip -c naive_S1_L001_R1_001.fastq.gz > naive_S1_L001_R1_001.fastq

I have seen bsub from LSF listed as equivalent to sbatch from SLURM, but so far it seems that they are only equivalent for submitting script files:

                  SLURM                    LSF
                  --------------------     ------------------
Job Submission    sbatch [script_file]     bsub [script_file]

Is there any other way to submit "one-liner" jobs with SLURM?

score 13 · Accepted Answer · edited Dec 21 '21 at 23:54

Try using the wrap option of sbatch. Something like the following:

ls *.gz | sed 's/.gz$//g' | xargs -I {}  sbatch --wrap="gunzip -c {}.gz > {}"

From the man page for `sbatch`:

--wrap=<command string>
       Sbatch will wrap the specified command string in  a  simple  "sh"  shell
       script,  and submit that script to the slurm controller.  When --wrap is
       used, a script name and arguments may not be specified  on  the  command
       line; instead the sbatch-generated wrapper script is used.

Holger Brandl · Answer 2 · 2016-05-10T13:02:07.087

4

You can also pipe into sbatch. Here's an example

echo '#!/bin/bash
touch hello_slurm.txt
' | sbatch -e err.log -o out.log

This could be "forced" into one line and also works well along with xargs -n1, but I think it is more readable this way to illustrate the idea.

Personally I prefer heredoc here, because it adds some more flexibility if the embedded "one-liner" or "some-liner" contains single quotes as well (which makes it imho also a more general solution compared to sbatch --wrap):

sbatch  -e err.log -o out.log <<"EOF"
#!/bin/bash
touch 'hello_slurm2.txt'
EOF

Btw, since it was mentioned in the question as well: the same approach works for bsub when using LSF.

edited May 10 '16 at 13:02

answered May 02 '16 at 16:10

Holger Brandl

10,634
3
64
63

The problem with a heredoc is that its difficult to use within a Makefile – user5359531 Dec 12 '18 at 21:40
Interesting. Perhaps use `echo -e ""` to build up a multiline script and then pipe it into `sbatch`... Thanks! – Christopher Bottoms Jun 23 '22 at 22:12

score 1 · Answer 3 · edited May 23 '17 at 12:00

Building on Carles Fenoy's answer, I created a utility called sbatch_run.

This script takes a job name and your command in quotes and then creates the script for you (and runs it for you).

sbatch_run jobname 'ls -lArt > list_of_files.txt'

Will create the following script and run it for you:

#!/bin/env bash
#SBATCH -J jobname.sbatch
#SBATCH -o jobname.sbatch.o_%j
#SBATCH -e jobname.sbatch.e_%j
#SBATCH --partition c14,general,HighMem
#SBATCH --mem 5G
#SBATCH --cpus-per-task 1
#SBATCH --nodes 1
#SBATCH --time 2-0

ls -lArt > list_of_files.txt

There are options for setting memory and cpus per task, etc.

Is there a "one-liner" for submitting many jobs to SLURM (similar to LSF)?

3 Answers3