Subprocess module in python claims 'command not found' for 'module list'?

Question

I am currently using the subprocess module in python for scripting purposes, and have been unable to execute the command 'module list' despite this working when I run it in shell and despite any other kind of command working fine when using the subprocess module to execute commands.

Two variations I have tried:

p = subprocess.Popen('module list', shell=True)
print(p.communicate()[0])

and

p = Popen(["module", "list"], stdout=subprocess.PIPE)
print(p.communicate()[0])

For commands such as 'echo hello world' and even longer commands with multiple arguments, either of these formats works just fine. Is the terminal I run commands from different from the shell used to execute commands using subprocess? The error I get is as follows:

/bin/bash: line 1: module: command not found

Welcome to SO! What does running `which module` in your shell give you? — Niayesh Isky, Apr 01 '19 at 19:45
Also, have you tried `print(subprocess.check_output('module list', shell=True))` or `print(subprocess.run('module list', shell=True, check=True, stdout=PIPE).stdout)` (depending on your Python version)? — Niayesh Isky, Apr 01 '19 at 19:52
Hey, thanks for your help. Using the alias 'which module' described in place of 'module' got the command working. — Farhana, Apr 01 '19 at 20:05
@NiayeshIsky actually so it doesn't seem to work as expected for 'module load'. When I run the command it outputs something like ''/tmp/modulescript_somenumber' — Farhana, Apr 01 '19 at 20:24
`which module` isn't an alias, it shows you where the `module` command is called from when it works in your shell. The fact that it's outputting a path to `/tmp` suggests that the method you used to set up Environment Modules could be non-standard. Could you [edit] your question and add how you installed Modules, as well as how you're importing module in Python? That should tell you why `which modules` is pointing in `/tmp`. — Niayesh Isky, Apr 02 '19 at 01:22
For example, I installed `environment-modules` on my Ubuntu distro using [this tutorial](https://askubuntu.com/a/533636/643673), and for me, `type module` tells me that `module` is a function, which means I can't call it from Python's subprocess directly. — Niayesh Isky, Apr 02 '19 at 01:26
Sorry, I meant that when I ran 'which module' it provided the location of the command that runs module, which i used in place of module and which worked successfully for module list (e.g. I replaced this with alias list) whereas when I did the same for module load (alias load), I got the tmp/modulescript output. — Farhana, Apr 02 '19 at 14:54

score 0 · Answer 1 · answered Apr 03 '19 at 15:35

Based on what you've said in the comments, I believe you're going about using environment modules in Python the wrong way: There is actually a method in Modules itself to import module functionality into Python, as explained here:

>>> execfile('/usr/local/Modules/default/init/python.py')
>>> module('list')
No Modulefiles Currently Loaded.
>>> module('load','foo')
>>> module('list')
Currently Loaded Modulefiles:
  1) foo/1.0

Of course, it's not very safe to use execfile(), so I slightly prefer the import method described here (slightly altered for Python 3 support):

import os

if 'PYTHONPATH' in os.environ:
    os.environ['PYTHONPATH'] +=':'+os.environ['MODULESHOME']+"/init"
else:
    os.environ['PYTHONPATH'] = os.environ['MODULESHOME']+"/init"

from python import module

score 0 · Answer 2 · answered Apr 07 '19 at 07:52

The documentation of the Environment Modules software provides a recommendation on how to initialize the module command in Python (that should work on either Python 2 or 3):

import os
exec(open('/usr/share/Modules/init/python.py').read())

Once initialized, the module function is available and could be used in the following way:

module('sub-command', 'arg1', 'arg2', ...)

For example:

module('load', 'foo', 'bar')
module('list')
module('avail')

Subprocess module in python claims 'command not found' for 'module list'?

2 Answers2