Eggs in path before PYTHONPATH environment variable

Question

If I have packages installed from easy_install, the eggs are prepended to sys.path before the items in the PYTHONPATH variable.

For example, if I have an egg package called foo installed as well as a package called foo in the current directory, and then do this:

PYTHONPATH="." python
>>> import foo

This will use the egg version of foo instead of the local directory. Inspecting sys.path shows that eggs are placed before items from PYTHONPATH. This seems broken. Is there any way to override this behavior?

Are you sure it won't pick up a package from the current directory? Normally the first entry in `sys.path` should always be the empty string i.e. it'll look in the current directory first. — kynan, Jan 12 '13 at 18:27
No, because the egg loader *prepends* its path, so it ends up before the current directory. — jterrace, Jan 12 '13 at 21:19
It prepends before the other *site* packages, but the first entry in `sys.path` is always `''` when running interactively, see [the docs](http://docs.python.org/2/library/sys.html?highlight=sys.path#sys.path). That's the case for me whatever the `PYTHONPATH`. — kynan, Jan 12 '13 at 23:40

samplebias · Accepted Answer · 2011-05-13T17:49:32.110

Unfortunately this is done with a hard-coded template deep inside setuptools/command/easy_install.py. You could create a patched setuptools with an edited template, but I've found no clean way to extend easy_install from the outside.

Each time easy_install runs it will regenerate the file easy_install.pth. Here is a quick script which you can run after easy_install, to remove the header and footer from easy_install.pth. You could create a wrapper shell script to run this immediately after easy_install:

#!/usr/bin/env python
import sys
path = sys.argv[1]
lines = open(path, 'rb').readlines()
if lines and 'import sys' in lines[0]:
    open(path, 'wb').write(''.join(lines[1:-1]) + '\n')

Example:

% easy_install gdata
% PYTHONPATH=xyz python -c 'import sys; print sys.path[:2]'
['', '/Users/pat/virt/lib/python2.6/site-packages/gdata-2.0.14-py2.6.egg']

% ./fix_path ~/virt/lib/python2.6/site-packages/easy_install.pth
% PYTHONPATH=xyz python -c 'import sys; print sys.path[:2]'
['', '/Users/pat/xyz']

For more clarification, here is the format of easy-install.pth:

import sys; sys.__plen = len(sys.path)
./gdata-2.0.14-py2.6.egg
import sys; new=sys.path[sys.__plen:]; del sys.path[sys.__plen:]; p=getattr(sys,'__egginsert',0); sys.path[p:p]=new; sys.__egginsert = p+len(new)

The two import sys lines are the culprit causing the eggs to appear at the start of the path. My script just removes those sys.path-munging lines.

It's really irritating that the user site doesn't have priority when paths/eggs are added to `sys.path`. On a machine where I don't have sudo I haven't found any way to override system-wide installed packages with user-installed ones. That looks like a pretty serious flaw in the way `sys.path` is assembled. — kynan, Jan 12 '13 at 18:40
I don't have enough rep to comment on the answer by samplebias, but here's a link to a bug report that gives the specific patch suggested by samplebias to fix this issue in setuptools: https://bugs.launchpad.net/ubuntu/+source/distribute/+bug/821000 — charlesreid1, Nov 20 '13 at 19:59

bsb · Answer 2 · 2017-08-18T09:17:06.223

Consider using the -S command-line option to suppress *.pth processing:

python -c 'import sys; print("\n".join(sys.path))'
python -S -c 'import sys; print("\n".join(sys.path))'

https://docs.python.org/3/library/site.html#site.main

You can also use -S with site.main() to delay *.pth processing until runtime, say to capture the original sys.path for appending:

export PYTHONPATH=$(
  PYTHONPATH='' \
  python -c 'import sys; \
    sys.path.extend(sys.argv[1:]); old=list(sys.path); \
    import site; site.main(); \
    [ old.append(p) for p in sys.path if p not in old ]; \
    sys.path=old; \
    print ":".join(sys.path)' \
  $EXTRA_PATH $ANOTHER_PATH)

python -S ... # using explicit PYTHONPATH

Start from explicit empty PYTHONPATH
Append to sys.path explicitly with extend
Import site and call site.main()
Append new paths to old path and then install it in sys.path
Print with ":" for PYTHONPATH
python -S is desirable for later runs only using $PYTHONPATH
python -S may or may not be desirable while setting PYTHONPATH (depending on if you need sys.path expanded before extending)

score 1 · Answer 3 · edited Feb 19 '13 at 18:57

I have done something like the following to prepend to the system path when running a top-level python executable file:

import sys
sys.path = ["<your python path>"] + sys.path

Often, the "<your python path>" for me involves use of the __file__ attribute to do relative look up for a path that includes the top-level module for my project. This is not recommended for use in producing, eggs, though I don't seem to mind the consequences. There may be another alternative to __file__.

Eggs in path before PYTHONPATH environment variable

3 Answers3

Linked