How to activate a specific Python environment as part of my submission to Slurm?

Question

I want to run a script on cluster (SBATCH file).

How can activate my virtual environment (path/to/env_name/bin/activate).
Does I need only to add the following code to my_script.sh file?

module load python/2.7.14 source "/pathto/Python_directory/ENV2.7_new/bin/activate"

Just run the `python` executable in the `bin` folder of the virtual environment. — Klaus D., Nov 29 '18 at 18:50
I think @KlausD. is suggesting you run something like `/path/to/miniconda3/envs/allennlp/bin/python -m allennlp.run --version`. — schmmd, Aug 27 '19 at 15:38
Would you mind posting a full example of the solution, please? — Kaleb Coberly, Feb 09 '23 at 02:53

score 17 · Answer 1 · answered Nov 29 '18 at 19:01

17

You mean to activate a specific Python environment as part of your submission to Slurm? This is what I add to my job script and it works well. Note that I use Anaconda, which by default adds the required paths to my .bashrc script after installation. Hope this helps.

....
# define and create a unique scratch directory
SCRATCH_DIRECTORY=/global/work/${USER}/kelp/${SLURM_JOBID}
mkdir -p ${SCRATCH_DIRECTORY}
cd ${SCRATCH_DIRECTORY}

# Activate Anaconda work environment for OpenDrift
source /home/${USER}/.bashrc
source activate MyEnvironment 

# we execute the job and time it
time mpirun python slurmscript.py

answered Nov 29 '18 at 19:01

Trond Kristiansen

2,379
23
48

what if my submission script is a python script? – Charlie Parker Nov 11 '20 at 17:22
You should be able to just execute your python script where I call mpirun. At that point the Anaconda environment is setup – Trond Kristiansen Nov 12 '20 at 00:15

score 2 · Accepted Answer · answered Nov 30 '18 at 14:55

2

Yes. Just make sure to insert those lines after the SBATCH ... lines and before any use of python.

answered Nov 30 '18 at 14:55

damienfrancois

52,978
9
96
110

1

what if my submission script **is** a python script? I can't just do that... – Charlie Parker Nov 11 '20 at 17:13
1

how do i create this environment on ALL nodes of my slurm cluster? – rohit_r Aug 25 '21 at 09:33
Which lines? Can you post a full example? – Kaleb Coberly Feb 09 '23 at 02:52

How to activate a specific Python environment as part of my submission to Slurm?

2 Answers2