6

I need to execute Python script in HPC cluster. Unfortunately, the default python version is just 2.6.6 and there is no numpy and scipy.

I can load these modules in command line

#module load /home/hw1u16/modules/2.7.3

and

module load /home/hw1u16/modules/1.6.2

However, when I write the bash script like this

module load /home/hw1u16/modules/2.7.3
module load /home/hw1u16/modules/1.6.2
python /home/hw1u16/project/trainAgent.py

It warns me

ModuleCmd_Load.c(200):ERROR:105: Unable to locate a modulefile for '/home/hw1u16/modules' ModuleCmd_Load.c(200):ERROR:105: Unable to locate a modulefile for '/home/hw1u16/modules' I don't know what's wrong, could any guys help me?

Xavier Delaruelle
  • 771
  • 7
  • 21
skwang
  • 61
  • 1
  • 1
  • 2
  • Please let me know if this works :) – DavidC. Jul 20 '17 at 20:33
  • The `/modules/` in `module load /home/hw1u16/modules/1.6.2` looks a little off. Are you sure it's not `module load /home/hw1u16/python-2.6` or something like that? – inspectorG4dget Jul 20 '17 at 20:37
  • Actually, I copy the file from the original file from the path shown in "module avail" command. I can execute like "module load python/2.7.3" on the shell terminal. But it's still useless in the bash script. Maybe I need some special setting? – skwang Jul 20 '17 at 21:01
  • @skwang see updated answer, please let me know – DavidC. Jul 20 '17 at 21:37
  • The error means there is no module file available in the path – Gang Jul 21 '17 at 00:37
  • I have found the fault. Coz I write the script in windows computer and scp it to the remote, function module cannot parse the path string formatted in windows. Thx all. – skwang Jul 21 '17 at 19:48

2 Answers2

9

I had a similar problem, and found two solutions:

  1. instead of running your script with sh yourscript.sh or ./yourscript.sh, you could run it as . yourscript.sh This will source the module correctly and run the script

  2. if you don't want to use . yourscript.sh, you can modify your shebang from #!/bin/sh to #!/bin/bash as noted in DavidC's answer and run your script as ./yourscript.sh Note that running it as sh yourscript.sh will not work

Paul Roub
  • 36,322
  • 27
  • 84
  • 93
Christian Seitz
  • 728
  • 6
  • 15
  • The shebang solution did not work for me (it also doesn't seem to be something DavidC said). The first solution worked great though. Didn't realise `.` was an alias for `source`. – Joshua Perrett Feb 09 '22 at 22:16
  • Same as @JoshuaPerrett, solution 1 works for me, solution 2 doesn't. – Carl Apr 04 '23 at 10:58
4

Okay, I think I know where is the problem. Try type module from the shell to see how module it is currently defined in your system. You will receive two options: either it is an alias or a function. This is because the module command is an alias or shell function.

Say your script is the following running.sh :

#!/bin/bash  
module load python/2.7.3
python /home/hw1u16/project/trainAgent.py

(It is a good practice to add the shebang)

To sort out this problem you have two options:

  • Option 1:

source the scitpt. In other words, do: source running.sh. This is exactly the same as typing the module command directly into your interactive shell. However, by doing ./running.sh, you are running a new, non-interactive shell. These generally do not have the standard aliases and shell functions set up.

  • Option 2:

Find the initialization script that defines the module command and source it from the script

DavidC.
  • 669
  • 8
  • 26
  • I have tried the `source` command, seems `module avail` can be use in bash script, but when I write `module load gcc/4.6.2` or `module load python/2.7.3`. It warns me `'ython(9):ERROR:105: Unable to locate a modulefile for 'python/2.7.3'. ` ` – skwang Jul 20 '17 at 22:12
  • @skwang This is the same error you reported before. Please, after writing the script as I wrote in the answer, _do_ `source running.sh` and then `./running.sh` in the same terminal – DavidC. Jul 20 '17 at 22:50