3

I am new to python and am learning things by writing scripts. I have tried the following and none of them seem to be working.

1) commands.getoutput('module load xxx')

2) subprocess.check_output(['module load', xxx'])

None of these change the environment as a side effect of the module call. Can someone tell me what is wrong?

h7r
  • 4,944
  • 2
  • 28
  • 31
pupil
  • 185
  • 9
  • By loading a module, do you mean `modprobe`? – h7r Feb 17 '15 at 21:54
  • I meant loading/switching of Environment modules . ex: module load gcc/3.1.1, module switch gcc gcc/3.2.0 etc – pupil Feb 18 '15 at 08:38
  • I am not familiar with this `module`. I found a reference for it on modules.sourceforge.net, but seems innaccessible. Can you give some reference for further help? – h7r Feb 18 '15 at 09:33
  • From what I read, I assume `module` just does changes to the shell's environment, so unless you start a subshell you won't have any of its effects. None of the variants you are using fit this need. – h7r Feb 18 '15 at 09:35
  • Yes module just changes the shell environment alone. As far as i see, when we change the shell environment, the current shell for which you have modified will have the new environment modules set. Can you tell me which method will work. I think this link will give you more details http://hpc.ucla.edu/hoffman2/computing/modules.php – pupil Feb 18 '15 at 10:21
  • The thing is that your methods don't create a shell, but call the process directly. You could even try something like `getoutput("bash -c module ...")` but this would only affect the shell created inside that call, not the shell form which you call the python script. I tend to believe that the effect you want can't be achieved from inside a python script. – h7r Feb 18 '15 at 10:30

3 Answers3

1

In case it helps anyone, I managed to do it by prefixing all commands that depend on the module with module load xxx &&. For example,

module load mpi/mpich && mpirun ./myprogram -n 4 -options

Not an elegant solution, but works for me. There's also this answer but I couldn't make it work in the system I have access to.

Aditya Kashi
  • 266
  • 2
  • 13
0

I found a different approach to solve this problem. I have written a shell script which load s the environment modules and i am calling that in the python script.

Something like this

import subprocess subprocess.call(['./module_load.sh'])

and the script has something like this....

module load ***

This seems to be working.

Have to say.....its pretty simple in Perl. it provides the environment modules package to handle this.

pupil
  • 185
  • 9
  • 1
    Out of curiosity: are you sure that after `subprocess.call` the environment available to the python process knows of the changes? Do you mind posting the effect of `os.getenv(...)` before and after this call, for some variable you know to be changed by `module`? – h7r Feb 18 '15 at 12:12
  • This does not load the modules outside the scope of the bash script. – SummerEla Jan 17 '17 at 05:13
0

Both commands create a subshell where the module is loaded - the problem is that this subshell is destroyed the moment Python function terminates

volcano
  • 3,578
  • 21
  • 28