0

I use slurm to run commands for timing experiments - the commands are stored in a text file and I submit array jobs to the scheduler. The problem is that some of the commands use quotes for arguments which include spaces - these quotes seem to get eaten up when run via the slurm batch system. I've tried to make a minimal example here to demonstrate.

The program that's run needs to take arguments, some of which contain spaces.

The dummy Java program I'm using for this example is:

class Hi {
    public static void main(String args[]){
        for (String a : args) { System.out.println("Arg is: " + a); }
    }  
}

The file containing the commands:

$ cat silly-commands.txt 
java Hi there friend
java Hi my friend "how goes it?"
java Hi my friend "how goes it?" blah

If I run the program using srun directly, I get the expected result:

$ srun -t 1 -n 1 java Hi my friend "how goes it?"
srun: job 23134019 queued and waiting for resources
srun: job 23134019 has been allocated resources
Arg is: my
Arg is: friend
Arg is: how goes it?

I run thousands of commands, so I use the array system. Here my basic slurm script:

#!/bin/bash
#
#SBATCH --job-name=dummy
#SBATCH --mail-type=FAIL
#SBATCH --mem=6gb
#SBATCH --output=logs/dummy_%A_%a.out
#SBATCH --error=logs/dummy_%A_%a.err
#SBATCH --account=dummy

module purge
module load lang/Java/13.0.2
CHUNKNAME=$1
LINE=`tail -n+${SLURM_ARRAY_TASK_ID} ${CHUNKNAME} |head -1`
srun -u -n1 $LINE

But now if I submit the job using the script, the commands get messed up:

sbatch -t 1 --array=1-3 simple-batch.sh silly-commands.txt 
Submitted batch job 23134028
$ more logs/dummy_23134028_2.out 
Arg is: my
Arg is: friend
Arg is: "how
Arg is: goes
Arg is: it?"

Any idea how to stop the command line being broken up and protect the arguments within the quotes?

UPDATE:

After comments here and links to other similar problems, I'm still unable to make it work, even when trying to use arrays.

I have tried:

CMD=($(head -${SLURM_ARRAY_TASK_ID} $1 | tail -1)) 
echo "Trying to run ${CMD[@]}"
srun -u -n1 ${CMD[@]}

and the output is:

$ cat logs/dummy_23146998_2.*                                                                  
Trying to run java Hi my friend "how goes it?"                                                                                      
Arg is: my                                                                                                                          
Arg is: friend                                                                                                                      
Arg is: "how                                                                                                                        
Arg is: goes                                                                                                                       
Arg is: it?"

So the third argument is still being broken up by the time java gets it. Am I trying the array thing correctly? I was trying to use this answer as advised .

Mr Felix U
  • 295
  • 1
  • 11
  • 2
    Quotes (and escapes and...) in variables' values are treated as plain characters, not as shell syntax. See ["Why does shell ignore quoting characters in arguments passed to it through variables?"](https://stackoverflow.com/questions/12136948/why-does-shell-ignore-quotes-in-arguments-passed-to-it-through-variables) and [BashFAQ #50: "I'm trying to put a command in a variable, but the complex cases always fail!"](http://mywiki.wooledge.org/BashFAQ/050) – Gordon Davisson Apr 28 '23 at 19:38
  • I have now been helped to get a solution by a colleague - the array stuff didn't help. What did help was calling `srun bash -c $CMD` to avoid any splitting of the arguments at all. It was buried in one of the suggested links, but not entirely obvious: https://stackoverflow.com/questions/12136948/why-does-shell-ignore-quoting-characters-in-arguments-passed-to-it-through-varia/46458483#46458483 – Mr Felix U Apr 30 '23 at 15:52

0 Answers0