Python easy_install in a virtualenv gives setuptools error

Question

There are a number of other StackOverflow questions similar to this one, but in each case, the platform was different or the error message was different or the solution had no effect or was outdated. I am trying to set up a Python 2.7.6 virtualenv and install modules into it but easy_install gives me errors indicating that setuptools is not available. But AFAIK easy_install is part of setuptools, so this makes no sense.

The problem only happens in a virtualenv. Here's what I've done:

• Created a brand new Red Hat 5 virtual machine
• Did a yum -y update to get the latest stuff, rebooted
• Downloaded Python-2.7.6.tar.gz, unzipped, ./configure; make; sudo make install
• Confirmed that python -V gives me 2.7.6 and sudo python -V also gives me 2.7.6
• wget https://bitbucket.org/pypa/setuptools/raw/bootstrap/ez_setup.py
• Modified ez_setup.py to add the --no-check-certificate flag to wget to get around proxy server problems in our network
• sudo python ez_setup.py
• sudo easy_install pip
• sudo pip install virtualenv
• virtualenv virtpy
• . virtpy/bin/activate
• easy_install elementtree

All of these steps succeed except for the last one, which fails with:

Traceback (most recent call last):
  File "/home/gperrow/virtpy/bin/easy_install", line 7, in <module>
    from setuptools.command.easy_install import main
  File "/home/gperrow/virtpy/lib/python2.7/site-packages/setuptools/command/easy_install.py", line 44, in <module>
    from setuptools.package_index import PackageIndex
  File "/home/gperrow/virtpy/lib/python2.7/site-packages/setuptools/package_index.py", line 203, in <module>
    sys.version[:3], require('setuptools')[0].version
  File "/usr/local/bin/scripts/pkg_resources.py", line 584, in require
    needed = self.resolve(parse_requirements(requirements))
  File "/usr/local/bin/scripts/pkg_resources.py", line 482, in resolve
    raise DistributionNotFound(req)  # XXX put more info here
pkg_resources.DistributionNotFound: setuptools

I'm starting with a clean VM and I've done nothing really unusual but I'm finding "easy_install" anything but. Am I doing something wrong or am I missing one or more steps?

Answer 1

I can't tell why exactly you get errors, but I am confident that there is a systematic approach that lets you cleanly install your custom Python including a working pip and virtualenv. In the following, I describe the procedure that I would use.

First of all, leave your system's Python untouched, for a number of reasons. One of them is that parts of your Linux distribution might depend on the specifics of its default Python. You don't want to break these parts. Another reason is that the vanilla Python installed to default locations might become confused by residuals of the original Python (distributions may have a specific Python/dist-packages/site-packages directory layout that differs from the vanilla one). This might or might not be a real problem in practice -- you can conceptually prevent these issues by not overwriting the system's Python. Another argument is that there is no need to install Python 2.7.6 as root. Install it as unprivileged user (called 'joe' from here on) and put it into /opt or something. This would be a clean start.

After having set up your custom Python, create yourself a little shell script, e.g. setup.sh that sets up the environment for using your custom Python version. Make sure to adjust and clean up the environment. Obviously, this especially affects PATH and PYTHONPATH. I would make sure that PYTHONPATH is unset and that PATH properly points to the custom install. Look at env and try to identify if there is anything left that might configure python in unexpected ways. After all, make sure that

$ command -v python
$ python -v

, executed as joe, look right.

Still being joe and under the proper environment, install pip for the custom Python. According to http://pip.readthedocs.org/en/latest/installing.html, download https://raw.github.com/pypa/pip/master/contrib/get-pip.py and execute it: python get-pip.py. Validate that it installed properly and that your environment is still right:

$ command -v pip
/CUSTOM/PYTHON/bin/pip

$ pip --version
pip 1.x.x from /CUSTOM/PYTHON/lib/python2.7/site-packages

At this point you should make sure that your environment does not contain any VIRTUALENV_* variables (which might have been set by your distribution or whatever component (unlikely, but worth checking)). If any VIRTUALENV_* variable is set, it most likely configures virtualenv in an unexpected way. Get rid of this (unset, or change). Then go ahead and install virtualenv into your new Python with the new pip, via pip install virtualenv. It might also be worth a try to install the latest development version of virtualenv via pip install https://github.com/pypa/virtualenv/tarball/develop.

Create and activate a new virtual environment. Using command -v pip, verify that pip comes from the virtual environment. Then install your custom package(s).

Note: I would definitely use pip to install things to the new virtual environment, and not easy_install, if possible. pip will quite soon be the official installer tool (it will be included with Python 3.4). If for some reason you really depend on easy_install, this should be possible (the easy_install command is provided by the virtual environment), but just to be sure you should also verify this via command -v easy_install.

Answer 2

Your approach is correct and other answers (Jan-Philip's and Piotr's) also, but your problem is simple:

You use a part of an old installation of setuptools on Python sys.path together with the new installation. It is obvious that the line numbers in pkg_resources.py should be by approximately by 100 lines greater for the current version of setuptools than it is in your traceback:

  ...
  File "/usr/local/bin/scripts/pkg_resources.py", line 669, in require  ## 584 is too old
    needed = self.resolve(parse_requirements(requirements))
  File "/usr/local/bin/scripts/pkg_resources.py", line 572, in resolve  ## 482 is too old
    raise DistributionNotFound(req)

The line numbers of the first three files of setuptools in traceback are correct: "virtpy/bin/easy_install", "virtpy/.../site-packages/setuptools/command/easy_install.py", "virtpy/.../site-packages/setuptools/package_index.py". Using different versions of the same package is everytimes a big problem.

Examine your Python sys.path by python -c "import sys; print sys.path" and think why "/home/gperrow/virtpy/lib/python2.7/site-packages" is not before "/usr/local/bin/scripts", or search everywhere for the string "/usr/local/bin/scripts". Fix it. One possible solution could be to install setuptools again locally into your active virtualenv: python ez_setup.py. (A fast method of examining the reason is to determine first if it is caused by user settings. Create a new user account, run the last three commands (virtualenv virtpy; ...) from that account, look at result and delete that user. If it works, examine which configuration file in your profile makes the problem.)

Verify finally that the new pkg_resources are used:

(virtpy)$ python -c "import pkg_resources; print pkg_resources"
<module 'pkg_resources' from '/home/you/virtpy/lib/python2.7/site-packages/pkg_resources.pyc'>
# not the obsoleted /usr/local/bin/scripts/...

Answer 3

I have a couple of suggestions, and also what I believe is your problem. Let's go for the problem first.

I noticed you said in your third bullet point

• Confirmed that python -V gives me 2.7.6 and sudo python -V also gives me 2.7.6

But you did NOT display the python version visible after the 2nd to last bullet point, when you activate your virtualenv. Since that step plays with your path, it's possibly not invoking the python you think.

What does python -V give AFTER you activate your virtualenv? I strongly suspect after the activate step you are being redirected and invoking the system python (which on RHEL is typically <= 2.5). It's important to RHEL that you not upgrade the system installed version of python, and the folks at RedHat go through several hoops to ensure this.

My first suggestion is to go back to the step where you installed python, and specify an alternate installation. Something like:

• ./configure --enable-shared --prefix=/opt/python2.7 && make && sudo make install

(note: --enable-shared is not specifically required...just a good idea)

The second suggestion I have is related to python package management. We will typically use easy_install to get pip installed. Once we have pip, we switch over to using pip for everything. It would be interesting to know what happens in your last step AFTER you activate your virtualenv, you then

• pip install elementtree

One more suggestion. After you've installed python2.7, and then installed virtualenv, pip & easy_install, you should then have *-2.7 version of these scripts available. It might be better to try invoking them & specify the version. This removes any ambiguity about the version you're requesting. eg:

• virtualenv-2.7 virtpy
• pip-2.7 install elementtree
• easy_install-2.7 elementtree

Answer 4

Did you try to use Software Collections? It is a standard approach on RHEL to get more up to date packages like python27:

http://developerblog.redhat.com/2013/08/08/software-collections-quickstart/

Then to use python27 you have to prefix all your python commands with

scl enable python 

e.g. to have a bash with python27:

scl enable python27 bash

This setup will possibly have a more compatible environment.