Python Subprocess / Environment Variable Script

Question

I have a compiled Fortran program (lets say in ~/codedir) linked against Intel MKL, which I want to run with a Python script in a certain directory (lets say calculationdir). I can navigate to calculationdir and execute ~/codedir/code, where code is the name of the executable. Everything works. But if I try to start the code from the below Python script:

import subprocess

command = '~/codedir/code'

with open('job.out', 'a') as f:
    process = subprocess.Popen(command.split(), stdout=f, shell=True)
    output, error = process.communicate()

I get the following error:

/home/codedir/code: error while loading shared libraries: libmkl_intel_lp64.so.1: cannot open shared object file: No such file or directory

I suspect that this might have something to do with the environment variables. I configured my shell that every time I start a shell, the script setvars.sh of Intel One API is executed which sets a lot of things. Could it be case that these variable are not set if I use python's subprocess? Do I have to tell subprocess in some way to execute also the setvars-script?

*Do I have to tell subprocess in some way to execute also the setvars-script?*. Most likely, yes you do have to. I'm a little surprised you didn't test that already. What shell are you using? — High Performance Mark, Apr 01 '22 at 10:41
Thanks for your comment. My question would then be how to do that? The documentation of the subprocess module is a bit confusing to me. I am using tilix. — Hendrik, Apr 01 '22 at 10:44
Did you `export` those variables? If you didn't, subpocesses will not see them. Set them before starting your Python script. — tripleee, Apr 02 '22 at 13:11

score 0 · Answer 1 · answered Apr 02 '22 at 13:04

You seem to be looking for

import subprocess
import shlex

with open('job.out', 'a') as f:
    subprocess.run(
        shlex.split(command),
        cwd=calculationdir,
        stdout=f, check=True)

There is no need to specify shell=True and in fact, when you split the command yourself, it's an error to do so on Unix-like platforms. (The semantics are slightly different on Windows, so while it's still an error, the symptoms are negligible.) See also Actual meaning of shell=True in subprocess

Plain split is wrong because it doesn't know how to cope with backslash escapes and quoting; if your command doesn't currently contain either of those constructs, you can get away with a basic string split; but the standard library supplies a function which works correctly regardless, and I see no reason not to use shlex.split here.

As the documentation explains, you should prefer subprocess.run over plain Popen when you can, as the latter requires you to copy/paste several lines of boilerplate code, and even then is less robust unless you know exactly what you are doing, probably at the expense of adding even more boilerplate code for error handling etc.

Finally, the actual beef: the keyword argument cwd allows you to set the working directory of the subprocess.

Python Subprocess / Environment Variable Script

1 Answers1