Install package dependencies with setup.py and wheels

Question

We're using an internally hosted PyPI server (devpi-server) so that we can host binary wheels of huge packages that take a long time to install from source like scipy, matplotlib, etc. Installing these packages with pip install scipy works perfectly and definitely uses the wheel we've created. However, working with any of our internally developed python packages that have a dependency on one of these packages and running python setup.py install|develop|test|whatever results in the following error:

No local packages or download links found for scipy
Traceback (most recent call last):
  File "setup.py", line 136, in <module>
    'develop': DevelopCommand
  File "/usr/local/lib/python2.7/distutils/core.py", line 112, in setup
    _setup_distribution = dist = klass(attrs)
  File "/users/me/virtualenvs/devpi-test/lib/python2.7/site-packages/setuptools/dist.py", line 239, in __init__
    self.fetch_build_eggs(attrs.pop('setup_requires'))
  File "/users/me/virtualenvs/devpi-test/lib/python2.7/site-packages/setuptools/dist.py", line 263, in fetch_build_eggs
    parse_requirements(requires), installer=self.fetch_build_egg
  File "/users/me/virtualenvs/devpi-test/lib/python2.7/site-packages/pkg_resources.py", line 564, in resolve
    dist = best[req.key] = env.best_match(req, self, installer)
  File "/users/me/virtualenvs/devpi-test/lib/python2.7/site-packages/pkg_resources.py", line 802, in best_match
    return self.obtain(req, installer) # try and download/install
  File "/users/me/virtualenvs/devpi-test/lib/python2.7/site-packages/pkg_resources.py", line 814, in obtain
    return installer(requirement)
  File "/users/me/virtualenvs/devpi-test/lib/python2.7/site-packages/setuptools/dist.py", line 313, in fetch_build_egg
    return cmd.easy_install(req)
  File "/users/me/virtualenvs/devpi-test/lib/python2.7/site-packages/setuptools/command/easy_install.py", line 587, in easy_install
    raise DistutilsError(msg)
distutils.errors.DistutilsError: Could not find suitable distribution for Requirement.parse('scipy')

And with easy_install:

$ easy_install scipy
Searching for scipy
Reading http://pypi.internal.example.com/us/base/+simple/scipy/
No local packages or download links found for scipy
error: Could not find suitable distribution for Requirement.parse('scipy')

If I grab the URL it's looking at I get:

$ curl http://pypi.internal.example.com/us/base/+simple/scipy/
<html>
  <head>
    <title>us/base: links for scipy</title></head>
  <body>
    <h1>us/base: links for scipy</h1>

    <form action="http://pypi.internal.example.com/us/base/+simple/scipy/refresh" method="post"><input name="refresh" type="submit" value="Refresh PyPI links"/></form>
us/external <a href="../../../external/+f/c48/5006bc28a8607/scipy-0.14.0-cp27-none-linux_x86_64.whl#md5=c485006bc28a8607b2fc1331df452dc1">scipy-0.14.0-cp27-none-linux_x86_64.whl</a><br/>
</body></html>

If I request the URL listed in that output, I get the wheel:

$ curl --silent 'http://pypi.internal.example.com/us/external/+f/c48/5006bc28a8607/scipy-0.14.0-cp27-none-linux_x86_64.whl#md5=c485006bc28a8607b2fc1331df452dc1' \
      | file -
/dev/stdin: Zip archive data, at least v2.0 to extract

By setting index_url in ~/.pip/pip.conf or passing -i INDEX_URL at the command line. — onlynone, Sep 09 '14 at 18:14
It looks like setuptools/distutils doesn't support installing wheels. Is this correct? It seems weird since they support creating wheels. — onlynone, Sep 19 '14 at 13:41
Distutils surely does not support that. Setuptools should though. Are you sure you are running the latest version? — rubik, Sep 19 '14 at 13:49
Looks like from within the source directory, where `setup.py` is, you can do `pip install .` and it will work. Pip will use the `whl`s from our internal PyPI repo, but plain setuptools/distutils won't. — onlynone, Nov 20 '14 at 15:53

score 1 · Accepted Answer · answered May 18 '18 at 20:07

First of all, the issue described in the question does not depend on using devpi and can be reproduced with any PyPI server, including the main repo at pypi.org. Example:

# setup.py
from setuptools import setup

setup(name='spam', install_requires=['vprof>0.37'])

(Instead of vprof, you can take any other package that does not ship anything besides wheels)

Test it:

$ pip install --upgrade "setuptools<38.2.0"
...
Successfully installed setuptools-38.1.0

$ python setup.py install
running install
running bdist_egg
running egg_info
writing spam.egg-info/PKG-INFO
...
Processing dependencies for spam==0.0.0
Searching for vprof>0.37
Reading https://pypi.python.org/simple/vprof/
No local packages or working download links found for vprof>0.37
error: Could not find suitable distribution for 
Requirement.parse('vprof>0.37')

Bang. Things get worse when you declare a binary-only package as a build dependency:

setup(name='spam', setup_requires=['vprof>0.37'])

Now all of the build and packaging commands will also fail, being unable to download build deps.

The issue solely depends on setuptools version used. Since 26 Nov 2017 and version 38.2.0, setuptools supports fetching and installing wheel dependencies, so if you still experience this issue:

upgrade `setuptools`

Newer OS releases should already ship a recent version of setuptools. For example, Ubuntu 18.04 has setuptools==39.0.1 by default (link). If you still have the old setuptools installed, most of the time it will be managed by the system package manager, so you shouldn't update it via pip. You can user-install an additional copy of setuptools

$ pip install --user --upgrade "setuptools>=38.2.0"

or use virtual environments for that matter.

Install package dependencies with setup.py and wheels

1 Answers1

upgrade setuptools

upgrade `setuptools`