5

In the user site .pth files are processed once at interpreter startup:

$ echo 'import sys; sys.stdout.write("hello world\n")' > ~/.local/lib/python3.8/site-packages/hello.pth
$ python3.8 -c ""
hello world

And it's the same behaviour in the system site, e.g. /usr/local/lib/python3.8/site-packages/.

But in venv they are processed twice:

$ rm ~/.local/lib/python3.8/site-packages/hello.pth
$ /usr/local/bin/python3.8 -m venv .venv
$ source .venv/bin/activate
(.venv) $ echo 'import sys; sys.stdout.write("hello world\n")' > .venv/lib/python3.8/site-packages/hello.pth
(.venv) $ python -c ""
hello world
hello world

Why are path configuration files processed twice in a virtual environment?

wim
  • 338,267
  • 99
  • 616
  • 750
  • Seems like there are still issues related to virtual environments (https://stackoverflow.com/questions/58805040/pywin32-226-and-virtual-environments). – CristiFati Feb 17 '20 at 14:06

1 Answers1

6

Looks like it all happens in the site module (not so surprising). In particular in the site.main() function.

The loading of the .pth files happens either in site.addsitepackages() or in site.addusersitepackages(), depending in which folder the file is placed. Well more precisely both these function call site.addpackage(), where it actually happens.

In your first example, outside a virtual environment, the file is placed in the directory for user site packages. So the console output happens when site.main() calls site.addusersitepackages().

In the second example, within a virtual environment, the file is placed in the virtual environment's own site packages directory. So the console output happens when site.main() calls site.addsitepackages() directly and also via site.venv() a couple of lines earlier that also calls site.addsitepackages() if it detects that the interpreter is running inside a virtual environment, i.e. if it finds a pyvenv.cfg file.

So in short: inside a virtual environment site.addsitepackages() runs twice.

As to what the intention is for this behavior, there is a note:

        # Doing this here ensures venv takes precedence over user-site
        addsitepackages(known_paths, [sys.prefix])

Which, from what I can tell, matters in case the virtual environment has been configured to allow system site packages.

Maybe it could have been solved differently so that the path configuration files are not loaded twice.

sinoroc
  • 18,409
  • 2
  • 39
  • 70
  • Good analysis. Looks like very hacky coding in `site.py`, I guess they don't care about [ab]uses of `.pth` files apart from the normal use-case of extending `sys.path` (which will be de-duplicated by `site.py` also). I advise you to permalink all these source references to git tag, e.g. [like this](https://github.com/python/cpython/blob/v3.8.0/Lib/site.py#L563) instead of [like this](https://github.com/python/cpython/blob/3.8/Lib/site.py#L563), so that they don't get lost when the 3.8 branch receives new commits. – wim Feb 12 '20 at 06:33
  • Yes, that's a good point about the links. Maybe the behavior is not intentional (loaded twice instead of just once), and it should be raised as an issue. – sinoroc Feb 12 '20 at 10:54