How does zsh interpret non-absolute paths in shebangs? (WAS: Why does python3 -i permit non-absolute paths in shebang?)

Question

I recently discovered the -i argument to Python, which drops into interactive mode after the script completes. Pretty neat!

$ cat test.py
#!python3 -i

x=5
print('The value of x is ' + str(x))
$ ./test.py
The value of x is 5
>>> print(str(x+1))
6
>>> 
zsh: suspended  ./test.py

However, when I tried to copy this script to a version that terminates on completion, it fails:

$ cat test1.py
#!python3

x=5
print('The value of x is ' + str(x))
$ ./test.py
/usr/local/Cellar/python3/3.6.3/Frameworks/Python.framework/Versions/3.6/Resources/Python.app/Contents/MacOS/Python: can't open file '
x=5
print('The value of x is ' + str(x))
': [Errno 2] No such file or directory

From some further reading, I discovered that I had originally made a mistake, and #!/usr/bin/env python3 is the correct shebang.

However, I'm curious why a non-absolute path to python3 succeeds only when I give the -i flag. I guess this must be something to do with how zsh interprets non-absolute shebangs, but I don't know enough to know how to investigate that.

System setup: MacOS 10.12.6, iTerm2 3.1.6, zsh 5.2. which python3 gives /usr/local/bin/python3, and that directory is on $PATH.

Interestingly, I don't get the same behaviour on sh:

$ sh
sh-3.2$ cat test.py
#!python3

x=5
print('The value of x is ' + str(x))
sh-3.2$ ./test.py
sh: ./test.py: python3: bad interpreter: No such file or directory

---

I got some comments suggesting that this is something to do with CWD or permissions. python3 is not in my CWD, and both files have execute permission:

$ ls -al | grep 'py' | awk '{print $1, $10}'
-rw------- .python_history
-rwxr-xr-x test.py
-rwxr-xr-x test1.py

Answer 1

Your kernel will not execute the script unless the interpreter is

• specified as an absolute path, or
• specified as a path relative to the current working directory

Then if the kernel refuses to execute the script, your shell might take over and try to execute it anyway, interpreting the shebang line according to its own rules (like finding the executable in the $PATH for example).

zsh does attempt to do this. sh does not.

However the way zsh interprets the shebang (and probably subsequent lines) is really really strange. It looks like it always expects a single argument after the command name. See what it does:

$ cat test.py 
#!python3 -b -i 

x=5
print('The value of x is ' + str(x))
$ strace -f -e execve zsh
execve("/bin/zsh", ["zsh"], 0x7ffd35c9e198 /* 78 vars */) = 0
host% ./test.py 
strace: Process 5510 attached
[pid  5510] execve("./test.py", ["./test.py"], 0x558ec6e46710 /* 79 vars */) = -1 ENOENT (No such file or directory)
[pid  5510] execve("/usr/bin/python3", ["python3", "-b -i", "./test.py"], 0x558ec6e46710 /* 79 vars */) = 0
[pid  5510] execve("/usr/lib/python-exec/python3.4/python3", ["/usr/lib/python-exec/python3.4/p"..., "-b -i", "./test.py"], 0x7fffd30eb208 /* 79 vars */) = 0
Unknown option: - 
usage: /usr/lib/python-exec/python3.4/python3 [option] ... [-c cmd | -m mod | file | -] [arg] ...
Try `python -h' for more information.
[pid  5510] +++ exited with 2 +++
--- SIGCHLD {si_signo=SIGCHLD, si_code=CLD_EXITED, si_pid=5510, si_uid=1000, si_status=2, si_utime=0, si_stime=0} ---
host% 
+++ exited with 2 +++

See how ["python3", "-b -i", "./test.py"] are passed as arguments. It seems highly counterintuitive to me to lump the two switches -b and -i together, but that's what zsh does. Python obviously doesn't understand this.

When there are no arguments, the exact behaviour depends on whether there is a space after the program name, but is strange in either case. Check it with strace yourself because you are not going to believe me.

It is my understanding that zsh handling of the shebang line is just buggy.