pip install --user error under OS X (virtualenv doesn't fix it)

Question

I used to install python (2.7.6) packages using sudo pip install [...] until I recently decided that this was a bad idea and I should use: pip install --user [...] instead.

Since I am using Mac OS X (10.10.3), my python modules had been installed into /Library/Python/2.7/site-packages. Now, with the --user option enabled, they are being installed into: /Users/USER/Library/Python/2.7/lib/python/site-packages

I have since moved all packages (including pip/setuptools) into this destination, adjusted my $PYTHONPATH and changed the file ownership to my local user. I expected that I would now be able to call pip install --user [..] and everything should go smoothly.

Unfortunately, this is not the case. For any pip install (or pip install --upgrade) command I receive the same error (below). The same is true when using a virtualenv - the same error shows up and I have no clue why.

However, everything does work "fine", when I use sudo pip install --user, which is why I assume that it is somehow related to a file-permission error I can not see. (It's not exactly fine, as the newly installed package will be in the right location, but all files are owned by root.)

Any idea what I am doing wrong or how to debug this?

pip install --upgrade --user tornado
Collecting tornado
  Using cached tornado-4.1.tar.gz
Exception:
Traceback (most recent call last):
  File "/Users/USER/Library/Python/2.7/lib/python/site-packages/pip/basecommand.py", line 223, in main
status = self.run(options, args)
  File "/Users/USER/Library/Python/2.7/lib/python/site-packages/pip/commands/install.py", line 280, in run
requirement_set.prepare_files(finder)
  File "/Users/USER/Library/Python/2.7/lib/python/site-packages/pip/req/req_set.py", line 317, in prepare_files
functools.partial(self._prepare_file, finder))
  File "/Users/USER/Library/Python/2.7/lib/python/site-packages/pip/req/req_set.py", line 304, in _walk_req_to_install
more_reqs = handler(req_to_install)
  File "/Users/USER/Library/Python/2.7/lib/python/site-packages/pip/req/req_set.py", line 483, in _prepare_file
abstract_dist.prep_for_dist()
  File "/Users/USER/Library/Python/2.7/lib/python/site-packages/pip/req/req_set.py", line 123, in prep_for_dist
self.req_to_install.run_egg_info()
  File "/Users/USER/Library/Python/2.7/lib/python/site-packages/pip/req/req_install.py", line 368, in run_egg_info
self.setup_py, self.name,
  File "/Users/USER/Library/Python/2.7/lib/python/site-packages/pip/req/req_install.py", line 339, in setup_py
import setuptools  # noqa
  File "/Users/USER/Library/Python/2.7/lib/python/site-packages/setuptools/__init__.py", line 11, in <module>
from setuptools.extension import Extension
  File "/Users/USER/Library/Python/2.7/lib/python/site-packages/setuptools/extension.py", line 8, in <module>
from .dist import _get_unpatched
  File "/Users/USER/Library/Python/2.7/lib/python/site-packages/setuptools/dist.py", line 21, in <module>
    packaging = pkg_resources.packaging
AttributeError: 'module' object has no attribute 'packaging'

I also checked the location of pkg_resources and deleted pkg_resources.pyc, which is located in /usr/local/lib/python2.7/site-packages for some reason, but it didn't change anything. (My local user also has full access rights in this location.)

[UPDATE:] For some reason I was able to install virtualenv without root (pip install --user virtualenv), but all other packages I tried still fail. Also, if I switch to a new virtual environment, I still receive the same error. I am now thinking this is related to my $PYTHONPATH variable, which appears to not be updated for my virtualenv, but I haven't found the real solution, yet.

I haven't used `pip` very much but I expect it knows if the package was installed system-wide or just for the user and that's what the error is. The solution would be to put things back as they were, record which packages were installed then uninstall and reinstall each package. — Droppy, May 26 '15 at 14:57
`pip` knows which packages are installed by looking through `$PYTHONPATH`. It doesn't know if this is a system-wide or user-specific installation. "Reverting" to the root library folder is not a solution, since you have to use `sudo` there without a doubt. The point of this is **not** to use `sudo` when installing new python packages. — Chris, May 26 '15 at 17:27
Ok, so I found this, but I don't agree with the accepted answer: http://stackoverflow.com/questions/15028648/is-it-acceptable-safe-to-run-pip-install-under-sudo?rq=1 — Chris, May 28 '15 at 11:35

score 0 · Accepted Answer · answered May 28 '15 at 12:15

Alright, so I found the actual problem and solution. My $PYTHONPATH contained an invalid directory. I setup my $PYTHONPATH a long time ago and wrote it into a .bashrc (or more accurately .bash_profile) script.

pip was probably looking through all directories in $PYTHONPATH, although I don't know what it is doing in each of them that causes the above error, but removing the invalid directory fixed my problems.

I also now highly recommend to just use virtualenv instead of a static setup.

pip install --user error under OS X (virtualenv doesn't fix it)

1 Answers1