1

I have multiple conda environments working fine. when I try to subit a job using qsub to SGE, the .bashrc file is not read. The batch script looks like:

#!/bin/bash
#$ -S /bin/bash
#$ -cwd
#$ -l h_cpu=48:00:00
#$ -l h_vmem=2048M
#$ -q pascal@pascal-[0123]-0[01234567]
## allocate the number of cores:
#$ -pe mpi 1
export OMP_NUM_THREADS=4

conda activate my_env
python ./test.py

I am getting the error: conda: Command not found. Also, it can't find bash commands like export.

source ~/.bashrc or source ~/.bash_profile does not help.

If I use qsub -V ...., the code somehow runs but it says

CommandNotFoundError: Your shell has not been properly configured to use 'conda activate'.
To initialize your shell, run

    $ conda init <SHELL_NAME>

Currently supported shells are:
  - bash
  - fish
  - tcsh
  - xonsh
  - zsh
  - powershell

See 'conda init --help' for more information and options.

IMPORTANT: You may need to close and restart your shell after running 'conda init'.

However, I do not want to use qsub -V because of some external reasons.

I am new to this cluster, so I am clueless. I just want to be able to choose freely between different conda enviroments in the batch script. Please help.

Edit 1: after seeing the comment

#!/bin/bash -l
#$ -cwd
#$ -l h_cpu=48:00:00
#$ -l h_vmem=2048M
#$ -q pascal@pascal-[0123]-0[01234567]
## allocate the number of cores:
#$ -pe mpi 1
setenv OMP_NUM_THREADS 1

conda activate my_env
python ./test.py

gives me conda: Command not found.

deltasata
  • 377
  • 1
  • 4
  • 21
  • As per duplicate, using `#!/bin/bash -l` for shebang should cover it. But I consider `conda run` preferable for programmatic execution in an environment context. E.g., `conda run -n my_env --live-stream python test.py` – merv Apr 14 '23 at 13:57
  • Neither work for me. – deltasata Apr 14 '23 at 14:17
  • @merv Neither works for me. – deltasata Apr 14 '23 at 14:18
  • Then your shell isn't initialized (see https://stackoverflow.com/a/55526573/570918). I.e., run `/full/path/to/bin/conda init bash` *once* in an interactive session. – merv Apr 14 '23 at 15:20
  • 1
    `conda init ...` within the batch script messed up all my previous envs, had to sort things manually, nightmare. What seems to work now: `export PATH="/path/to/conda/bin:$PATH"` `eval "$(conda shell.bash hook)"` and then `conda activate my_env` – deltasata Apr 14 '23 at 15:36
  • More fundamentally, understand that `qsub` jobs do not run an interactive shell. If there are shell setup routines you want to use from both interactive and noninteractive shell instances, probably move those to a separate file which you `source` both from your `.bash_profile` and from any script which needs those settings. This is a common FAQ. – tripleee Apr 14 '23 at 15:49
  • On PBS jobs, I could switch between conda envs easily within the script. – deltasata Apr 14 '23 at 16:01
  • No one should recommend to run `conda init` in a batch script. Run once per user in interactive session...done. Clean up the user `.bashrc` (or `.bash_profile`) if you encounter issues. – merv Apr 14 '23 at 16:30
  • @tripleee does `qsub` not respect shebang `bash -l`? LSF and SLURM certainly do. – merv Apr 14 '23 at 16:31
  • Cleaner is `eval "$(/path/to/conda/bin/conda shell.bash hook)"`. No need to manually add to PATH. – merv Apr 14 '23 at 16:50
  • `bash -l` runs a login shell, which reads `.bash_profile`, not `.bashrc`. It's not clear from the exposition which of these files contains whatever `conda` initialization the OP requires. – tripleee Apr 14 '23 at 16:59
  • my .bashrc has the conda initialization part – deltasata Apr 14 '23 at 17:32
  • So, probably `bash -i`; but again, forcing an interactive shell for a noniteractive task is a pretty crude workaround, and it's probably better to isolate those parts into a separate file you `source` from both places; or indeed, simply copy the `conda` initialization code into scripts which need it. – tripleee Apr 14 '23 at 20:05

0 Answers0