2

I wanted to run a python script with sbatch, however, it seems that the only way to run a python script with sbatch is to have a bash script that then run the python script. As in having batch_main.sh:

#!/bin/bash
#SBATCH --job-name=python_script

arg=argument

python python_batch_script.sh

then running:

sbatch batch_main.sh

The issue with this is that I'd wish to have a separate config file for the arguments (since its usually not a single number or argument) and also be able to use the array option. Also, I usually run multiple different sbatch jobs simultaneously (with different configurations), thus it would be nice if changing the configuration file does not make the different sbatch runs get in the way with one another (since if the job gets queued and then the config file changes later, it will run the most recent config file rather than the copy of the config when I ran sbatch). To get around this issue I discovered that when I run a sbatch script, SLURM actually copies the submission script to its internal database (I disovered it after asking: Changing the bash script sent to sbatch in slurm during run a bad idea?). Therefore, I actually decided to hard code the configuration into the bash submission script (making the submission script the config file essentially). That way I just edit the submission script and then run the file. However, I'd like to stop this since this makes me write in bash which I want to avoid at all cost.

Ideally, I'd like to run a sbatch that directly runs python.

Since this might not be possible, I wanted to know what other options did exist to solve this issue. For example, is it possible to have slurm copy a different file (like a python config file) to its internal database so that when it queues the job it runs the job that I exactly want to run? (notice that running a sbatch job and then changing the config file is not the way to do this, since this might cause issue when one changes the config file, slrum will read the most recent copy of the config rather than the copy of the config when the job was ran). Or what other options do I have? Am I really stuck with writing bash or can I do something else to deal with configurations in python rather than some other weird hack?

In general I also wanted to know what people did for this in the real world or what a good practice/standard was for this.

Community
  • 1
  • 1
Charlie Parker
  • 5,884
  • 57
  • 198
  • 323
  • 2
    You know that you can change the shebang line, right? For example, `#!/bin/env python` is a valid shebang line for `sbatch` scripts, assuming that the script is written in python. – Christopher Bottoms Feb 01 '17 at 20:35

1 Answers1

5

Python scripts are valid submission files provided that they start with the python shebang (typically #!/usr/bin/env python).

For instance: 

#!/usr/bin/env python
#SBATCH --time=...
#SBATCH --partition=...

import sys
jobid=sys.environ["SLURM_JOB_ID"]
print "Hello World from job %s" % jobid

Note that if your script imports custom modules, you will need to set the PYTHONPATH even if they lie in the current directory.

damienfrancois
  • 52,978
  • 9
  • 96
  • 110