2

Background

Below, I detail two different ways of running Python files - I am confused about the difference between them.

  1. Running the Python file as an executable.

To run a Python file as an executable, I must first set a shebang in my file (# /usr/bin/env python3), then run $ chmod +x filename.py at the command line, then run $ ./filename.py at the command line to execute the file.

  1. Running the Python file through the python3 command line command.

To run a Python file through the python3 command, I open my command line and run $ python3 filename.py.

My Question

I understand that, when running the Python file as an executable, the shebang directs the computer to launch the (in this case) python3 interpreter which will interpret the subsequent code in the file and therefore run the file. When running the file through the python3 command, I understand that this is just another way of directing the computer to launch python3 to interpret the code in the file. To me, these two techniques therefore seem identical.

Am I missing something? What's the difference, if any, between these two ways of running a Python file?

zen_of_python
  • 157
  • 2
  • 10

3 Answers3

2

In practice, they're identical.

The shebang is just a convention that tells the OS what to do with the script file that you've chmod-ed to be executable -- i.e., executable with what. Without it, the OS just treats the file as a text file and will try to execute it as a shell script.

Matt Howell
  • 15,750
  • 7
  • 49
  • 56
  • 2
    It is not actually true that without the shebang the OS will treat the script as a shell script. The kernel will not know what to do with it at all if you try to execute it (rather than give it as command line argument to an interpreter). If you are invoking it **from a shell** then the shell will fall back to treating it as a shell script if the exec fails, but this is behaviour of the shell, not the kernel; if for example you try it from python using `subprocess.Popen` then you will see `OSError: [Errno 8] Exec format error`. – alani Aug 23 '20 at 05:13
  • I was indeed assuming the context was invoking from the shell. – Matt Howell Aug 23 '20 at 06:05
2

From the point of view of the system the shebang line is necessary when the file is run as an executable. After checking the permission bits the OS sends the file to the program loader which determines how to run the program by parsing the first line as an interpreter directive. Based on this the loader executes python3 based on the specified path to the executable. If instead /usr/bin/env was used in shebang line then the env command finds the executable based on PATH ( env can also be used to make other modifications to the environment using NAME=VALUE pairs). The loader then passes to the program the path that was used when the user tried to run the script as an argument.

In the second case the OS just loads python3 and passes the script as an argument, it doesn't care at all about the file and its permissions.

From the point of view of the user (which IMO is more important) the shebang line is just another level of abstraction that hides the details of implementation (in this case what program is used to run the script). This means the line can be changed (perhaps by modifying the environment or using a different executable) without the user having to change the way they invoke the script. Also, if the user puts the script in a location that is in PATH then they can invoke the script from anywhere without first navigating to the directory or remembering the location.

On the other hand invoking python3 directly allows the user to pass additional flags to python3 for example -i for interactive runs and -m to use additional modules such as pdb for debugging.

Edit: Based on @alaniwi's comment below explained the role of env in finding the path of the python3 executable in more detail.

SigmaPiEpsilon
  • 678
  • 5
  • 15
  • I've given you upvote, but just to pick up on your "modified environment" point - although `env` has the *potential* to modify the environment (hence the name), that is not what it is being used for here. It is a trick to find `python3` via a `PATH` lookup rather than have to hard-code the full path to the interpreter (bearing in mind that the shebang requires a full path). `env` itself is reliably to be found in `/usr/bin`, but `python3` could be anywhere, especially if a virtual environment has been activated. – alani Aug 23 '20 at 06:23
  • @alaniwi Thanks for the upvote and the interesting point. AFAIK `env` passes the specified environment to the command (by using `NAME=VALUE` pairs). In the default case when no such pair is included it passes the current environment unchanged. Finding the right `python3` is just a side effect as `PATH` is one of the environment variables that are passed and it is used by shell to find the location of `python3` (the first one that appears in `PATH`). – SigmaPiEpsilon Aug 23 '20 at 06:34
  • 1
    No shell is involved. `env` itself uses `PATH` to find the executable. That is the whole point. (Yes it will also pass `PATH` as one of the environment variables to `python3`, but that does not require any special action by `env`; child processes inherit the parent process's environment variables automatically.) – alani Aug 23 '20 at 06:48
  • @alaniwi What I was trying to say is that `env` uses the path you specify after it to find the executable. If nothing is specified then it uses the currrent `PATH`. For example if you invoke python as `$ env PATH=$SOMEPATH python3` then the first python found in `SOMEPATH` is invoked rather than the first python in current `PATH`. Anyway, I have modified the post a bit to highlight the path finding function of `env`. – SigmaPiEpsilon Aug 23 '20 at 07:29
  • Adding extra environment variables in the shebang is not used in this question and I haven't seen it used, and indeed, if I try it then I get an infinite loop when I try to execute the script -- I haven't worked out exactly why. – alani Aug 23 '20 at 07:50
  • I've finally worked out why the infinite loop happens. If `test.py` has `#!/usr/bin/env FOO=bar python` then when you execute `./test.py` the `FOO=bar python` is not split by space but treated as a single argument to `env` as if you had typed at the shell prompt: `/usr/bin/env 'FOO=bar python' ./test.py` - i.e. sets the variable `FOO` to the value `bar python` and execute `./test.py` again (instead of `python ./test.py`), hence the infinite loop. – alani Aug 23 '20 at 08:13
  • 1
    @alaniwi Yeah this is Linux specific thing, in Mac and some BSDs you can pass environment variables in shebang. Although in newer version of coreutils (env version 8.3) it allows you to include a [`-S` flag](https://www.gnu.org/software/coreutils/manual/html_node/env-invocation.html#g_t_002dS_002f_002d_002dsplit_002dstring-usage-in-scripts) that can do the string splitting and hence once can modify environment and also pass different options to the command. – SigmaPiEpsilon Aug 23 '20 at 08:33
  • That's handy to know - thanks. I now see that it is mentioned at https://stackoverflow.com/a/52979955/13596037 – alani Aug 23 '20 at 08:41
1

Nope, you have pretty much captured it.

A practical consequence is that the shebang relieves you from having to remember whether it's python3 frobnicate or python frobnicate or sh frobnicate or bash frobnicate or awk frobnicate or perl frobnicate or...

This also makes it easy down the line to change your mind. Many tools of mine have started life as simple shell scripts, then been rewritten in Python or something else; but the calling interface doesn't change.

Before Unix, there was an unbridgable gap between system utilities (which you invoke simply by name) and user scripts (which before the introduction of the shebang always had to be called with an explicit interpreter).You still see remnants of this division in lesser systems. An important consequence was that users were able to easily and transparently wrap or replace standard commands with their own versions. This in some sense democratized the system, and empowered users to try out and evaluate improvement ideas for the system on their own. (Figuring out why your brilliant theory wasn't so great in practice is also an excellent way to learn and improve.) I don't think the importance of this versatility and flexibility can be overstated; it's one of those things which converted us from mere users to enthusiasts.

tripleee
  • 175,061
  • 34
  • 275
  • 318
  • I'm not *that* old but this was definitely still a distinguishing feature of Unix in the late 1980s. – tripleee Aug 23 '20 at 06:46