Pip ignores dependency_links in setup.py despite proper format

Question

Looking as solutions in the past such as pip ignores dependency_links in setup.py, this configuration should work.

Relevant content of my setup.py

packages=find_packages(),
dependency_links=['http://github.com/koji-project/koji/tarball/master#egg=koji'],
install_requires=['jira', 'PyYAML', 'requests', 'psycopg2',
                  'elasticsearch', 'beanbag', 'pyzabbix', 'enum34',
                  'beautifulsoup4', 'pytz', 'koji'],
tests_require=['flake8', 'autopep8', 'mock'],
include_package_data=True,
cmdclass={'test': setupTestRequirements}

The only thing I can think of is that my url is invalid. I don't see why it would be since it is of version 1.14.0.

Upon running pip install . I get.

Could not find a version that satisfies the requirement koji (from MARs==0.17.10) (from versions: ) No matching distribution found for koji (from MARs==0.17.10)

Upon running python setup.py develop --user, the output doesn't mention Koji

Answer 1

Your configuration is correct. However the problem lies elsewhere. Take a look at the koji repo on github: the project has no setup.py committed. As long as there's no setup.py script, neither pip nor setuptools (via setup.py install/setup.py develop) won't be able to install your project because they won't be able to install koji dependency as it is no valid python package at all.

Update:

The problem with koji repo on github is that it is only a mirror of the actual dev repo located on Fedora Pagure and is not synced with the upstream. So the correct answer is to use the real development repository instead of the github mirror:

dependency_links=['git+https://pagure.io/koji.git#egg=koji-1.14.0']

Easy peasy. :-)

Original answer (obsolete, only if you want to install from kojis repo mirror on Github):

I see two ways out of this situation:

Forking

1. fork koji on github
2. write your own setup.py script or copy it somewhere (see below for more info), commit and push
3. adapt the URL in dependency_links in your project's setup.py.

For testing, I prepared a fork of koji with a setup script; if I use its URL instead of the upstream repo, the installation succeeds. I also tagged my own "release" with koji-1.14.0.post1 to distinct the version with the setup script from the vanilla ones. Example setup.py with the new dependency:

from setuptools import setup, find_packages

setup(
    name='spam',
    version='0.1',
    author='nobody',
    author_email='nobody@nowhere.com',
    url='www.example.com',
    packages=[],
    dependency_links=['https://github.com/hoefling/koji/tarball/master#egg=koji-1.14.0.post1'],
    install_requires=['koji==1.14.0.post1'],
)

Testing the installation with pip yields:

$ pip install . --process-dependency-links
Obtaining file:///home/hoefling/python/spam
  DEPRECATION: Dependency Links processing has been deprecated and will be removed in a future release.
Collecting koji==1.14.0.post1 (from spam==0.1)
  Downloading https://github.com/hoefling/koji/tarball/master (1.4MB)
    100% |████████████████████████████████| 1.4MB 759kB/s 
Collecting pyOpenSSL (from koji==1.14.0.post1->spam==0.1)
  Using cached pyOpenSSL-17.5.0-py2.py3-none-any.whl
Collecting pycurl (from koji==1.14.0.post1->spam==0.1)
  Using cached pycurl-7.43.0.1.tar.gz
...
Installing collected packages: six, idna, asn1crypto, pycparser, cffi, 
cryptography, pyOpenSSL, pycurl, python-dateutil, chardet, certifi, 
urllib3, requests, pykerberos, requests-kerberos, rpm-py-installer, 
koji, spam
  Running setup.py install for rpm-py-installer ... done
  Running setup.py install for koji ... done
  Running setup.py install for spam ... done
Successfully installed asn1crypto-0.23.0 certifi-2017.11.5 cffi-1.11.2 
chardet-3.0.4 cryptography-2.1.4 idna-2.6 koji-1.14.0.post1 pyOpenSSL-17.5.0 
pycparser-2.18 pycurl-7.43.0.1 pykerberos-1.1.14 python-dateutil-2.6.1 
requests-2.18.4 requests-kerberos-0.11.0 rpm-py-installer-0.5.0 six-1.11.0 
spam-0.1 urllib3-1.22

Installed packages look good:

$ pip list
Package           Version     
----------------- ------------
asn1crypto        0.23.0      
certifi           2017.11.5   
cffi              1.11.2      
chardet           3.0.4       
cryptography      2.1.4       
idna              2.6         
koji              1.14.0.post1
pip               9.0.1       
pycparser         2.18        
pycurl            7.43.0.1    
pykerberos        1.1.14      
pyOpenSSL         17.5.0      
python-dateutil   2.6.1       
requests          2.18.4      
requests-kerberos 0.11.0      
rpm-py-installer  0.5.0       
rpm-python        4.11.3      
setuptools        38.2.4      
six               1.11.0      
spam              0.1         
urllib3           1.22        
wheel             0.30.0

The downside of this method is the additional overhead you get in maintaining the fork until the setup script is merged into upstream. This includes testing and eventually adapting koji's setup.py in your fork each time you want to sync the upstream updates. I would probably create a separate branch with the setup script committed there, sync the fork as usual and then rebase the branch on top of fork's master, but if you are used to another update strategy, stick to it.

Use koji package from TestPyPI

Actually, I found some koji wheels of the most recent version on TestPyPI. This is also the place where I got the setup.py for the fork above - I downloaded the source tar, unpacked it and copied the setup script. This means that the koji devs are looking into distributing the project via PyPI and are working on the setup script, but didn't commit it yet. While they are working on it, you can use the testing package index as the workaround. This way, you will not build the package from sources, taking the wheel instead that koji devs built and uploaded:

setup(
    ...
    dependency_links=['https://testpypi.python.org/pypi/koji'],
    install_requires=['koji'],
)

The downsides of this method are:

1. You don't know if the koji package from TestPyPI is installable at all. Even if it is, there's no guarantee that the installed code will work as intended (although it should). When you have the fork, you can always fix the setup script yourself - here you are doomed if the wheel file has errors.
2. Packages on TestPyPI are removed on a regular basis. From the docs:

   Note: The database for TestPyPI may be periodically pruned, so it is not unusual for user accounts to be deleted.

Last note

You can of course combine the two workarounds and use both URLs in dependency_links:

setup(
    ...
    dependency_links=[
        'https://testpypi.python.org/pypi/koji',
        'https://github.com/hoefling/tarball/master#egg=koji-1.14.0.post1',
    ],
    install_requires=['koji'],
)

This way, if the package is not found on TestPyPI, it will be built from your fork.

Last note 2

You will probably need to install some additional system packages; at least for my system CentOS Linux release 7.3.1611 (Core) I had to install curl-devel to satisfy pycurl.