Why is subprocess ignoring PATH, and how can I change this?

Question

I need to change which program is called by a Python application. Unfortunately I cannot change the Python code. I can only change the calling environment (in particular, PATH). But unfortunately Python’s subprocess module seems to ignore PATH (at least under certain circumstances).

How can I force Python to respect PATH when searching which binary to invoke?

To illustrate the problem, here’s an MVCE. The actual Python application is using subprocess.check_output(['nvidia-smi', '-L']), but the following simplified code shows the same behaviour.

Create test.py:

import os
from subprocess import run

run(['which', 'whoami'])
run(['/usr/bin/env', 'whoami'])
run(['whoami'])

os.execvp('whoami', ['whoami'])

Now create a local whoami script and execute test.py:

echo 'echo foobar' >whoami
chmod +x whoami
PATH=.:$PATH python3 test.py

On my system¹ this prints:

./whoami
foobar
konrad
konrad

I expect this code to always print foobar instead of konrad.

My MVCE includes the os.execvp call because the subprocess documentation states that

On POSIX, the class uses os.execvp()-like behavior to execute the child program.

Needless to say, the actual execvp POSIX API, called from C, does respect PATH, so this is a Python specific issue.

---

¹ Ubuntu 18.04.2 LTS, Python 3.6.9.

Answer 1

as per my comment, this is due to Python's implementation of execvp not being consistent with POSIX execvp semantics. in particular Python doesn't respond to ENOEXEC errors by interpreting the file as a shell script and needs an explicit shebang.

creating the file as:

printf '#!/bin/sh\necho foobar\n' > ./whoami

causes things to work as expected

note that this has been known about for a while: https://bugs.python.org/issue19948

Answer 2

Your

echo 'echo foobar' >whoami
chmod +x whoami

is not running correctly.

Python is not picking it up a executable, even though though the execute bit is set, it doesn't know it needs to run bash first, to execute, thus is skips in on the path and runs the original whoami which is pathed /usr/bin/whoami

adding shebang

echo "#!/bin/sh" > whoami
echo 'echo foobar' >> whoami
chmod +x whoami

On Unix-style systems (including Linux/OS X), the shebang line—as it’s called—tells the loader (or kernel, or occasionally shell) which program to use to run the file. At its most basic, you’d specify a path to the python interpreter.

i suspect if you ./whoami (with execute permission set), the shell is doing some extra magic so you don't have to type /bin/sh $PWD/whoami

if you do

chmod -x whoami

you can use the special

. ./whoami (tell the shell to execute it as a shell script).

note that execvp should use /bin/sh rather than bash. also . ./whoami will depend on what shell you happen to be using, most will "source" the file rather than executing it in another process (i.e. changes to environment, working directory, etc will remain)

If no shebang or executable header, the shell simply uses itself as the default interpreter (but only when invoked via ./whoami; . ./whoami is different, it sources the file, regardless of whether it’s executable).

The confusing nature of execvp (POSIX, not Python) apparently also does this. Python fails in this case because os.execvp actually does not call execvp under the hood, it similarities are just in name.