25

Is there a way for a python script to load and use environment modules? os.system('module load xxx') doesn't work since it executes them in a subshell (at least, I think that's what's happening).

Pete
  • 10,310
  • 7
  • 53
  • 59
marshall.ward
  • 6,758
  • 8
  • 35
  • 50

5 Answers5

22

I know this question's kind of old but it's still relevant enough that I was looking for the answer, so I'm posting what I found that works as well:

At least in the 3.2.9+ sources, you can include the python "init" file to get a python function version of module:

>>> exec(open('/usr/local/Modules/default/init/python.py').read())
>>> module('list')
No Modulefiles Currently Loaded.
>>> module('load','foo')
>>> module('list')
Currently Loaded Modulefiles:
  1) foo/1.0

I've been told earlier versions can do the same without the .py extension, but that's second hand, so ymmv.

Alternative "init" file location (from comment by @lib): /usr/share/Modules/init/python.py

To use with Python 3, version 4.0 or later of Environment Modules is required, as that is the first version to have a bug-free Python3-compliant version of the Python init file.

jnewman
  • 377
  • 2
  • 10
  • This is much closer to what we actually use (since the `popen` calls to `modulecmd` are in the `python.py` file that you cited), so I'll set this to the answer. Thanks! – marshall.ward Aug 20 '13 at 01:07
  • I had nearly a heart attack figuring this out! Thank you very much. – yvesonline Oct 30 '13 at 15:53
  • 6
    In the server I am using it is in /usr/share/Modules/init/python.py . I found the path be running module --version (the correct entry seems MODULES_INIT_DIR) – lib Feb 13 '15 at 07:54
  • 1
    Cool, I found this answer looking for a solution for perl, and my particular setup (VERSION 3.2.7 on RHEL 6.2) has initialization code for several shells in /usr/share/Modules/init/: bash, csh, ksh, perl, python, sh, tcsh, and zsh. Thanks for the pointer. – Dave X Dec 16 '16 at 15:30
  • 1
    execfile() is deprecated in python 3.6.0. How to use exec statement with python 3.6 to use environment modules? – srand9 Jan 30 '17 at 08:30
  • 1
    Thanks @Jarvis. The answer has been updated for Python 3 – jnewman Jun 12 '19 at 14:10
4

One of our admins was able to solve the problem for me using os.popen() calls to modulecmd:

cmd = os.popen('/path/to/modulecmd python load my-module')
exec(cmd)
marshall.ward
  • 6,758
  • 8
  • 35
  • 50
2

While the accepted solution works, I found it to be easier to write:

import sys
sys.path.insert(0, '/path/to/environment/modules')
# Environment modules become available by loading python.py (the name choice could be better here)
from python import module

# ...
module('use', '/some/dir')
module('load', 'program/1.2.3')

This looks more pythonic to me and also it allows IDEs to offer auto-completion etc.

ian
  • 674
  • 1
  • 6
  • 11
andreee
  • 4,459
  • 22
  • 42
0

Not directly, but here's one possible workaround, depending on your environment. Assuming you can preface your system command with ENVVAR=value, you can do something along these lines:

import os
os.environ['EDITOR'] = 'vi'
cmd = "EDITOR=%(EDITOR)s $EDITOR" % os.environ
os.system(cmd)

The code assigns vi to your EDITOR environment variable, then passes it on the command line and runs the command, which (in this case) is EDITOR.

Jeff Bauer
  • 13,890
  • 9
  • 51
  • 73
0

I found the answers from jnewman, andreee and ian quite helpful. However, in my case the $MODULESHOME/init/python.py is written in Python 2 and is incompatible with Python 3 (it uses exec output instead of exec(output)). Here is my modified version that supports Python 3:

import os
import subprocess

def load_module(module_name, *, capture_output=True, check=True, **kw):
    module_home = os.environ["MODULESHOME"]
    modulecmd = os.path.join(module_home, "bin/modulecmd")
    process = subprocess.run(
        [modulecmd, "python", "load", module_name],
        capture_output=capture_output,
        check=check,
        **kw,
    )
    return process

Example use case:

CONDA_MODULE = "Anaconda3/2022.05"
load_module(CONDA_MODULE)
print(subprocess.check_output(["conda", "--help"], stderr=subprocess.STDOUT).decode("utf8"))
Silverstone
  • 11
  • 3