Heroku Django setup install_requires from github

Question

I have a heroku project that is trying to install a Django reusable app using pip install -r requirements.txt (it's not the exact command from the official heroku/python buildpack, but it's basically doing that).

The problem is that my Django reusable app has a requirement itself, from a Github public repository.

A few answers in pip install dependency links point to:

• The dependency-links syntax being not compatible with pip 19
• The new syntax (i.e. package @ git+protocol://...) of course not being compatible with older versions of pip.

My Django project can be deployed in different ways: Heroku, local, Docker (dev and prod). Most of these deployment methods have the latest version of pip.

Heroku's python buildpack has Pip 9.0.2

My other deployment methods have Pip 19, so they have incompatibilities in the way they read requirements, and I can't have a method that works for both.

For older versions of pip, I do in my setup.py:

setup(
# [...]
install_requires=[
        "my_package == x.y.z"
    ],
    dependency_links=["git+https://github.com/account/package/tarball/master#egg=my_package-x.y.z"]
)

Whereas for Pip 19, I do:

setup(
# [...]
install_requires=[
        "my_package @ git+https://github.com/company/package.git"
    ],
)

I tried forking the repo and just increasing the version to pip's latest version. It does work, except for errors in pip-diff, which don't seem to have an effect on the build.

Interesting to note too is that since the buildpack has a cache system, you generally notice this problem only once until you require a newer version of a library. But it's a problem with review apps for example.

So what is the solution here? I could:

1. Downgrade all other deployment methods to an older version of pip that works with dependency links, including all my local dev setups
2. Use a forked version of heroku's python buildpack, with updated pip (pip-diff needs to be fixed though)
3. Remove dependencies from github and add those dependencies at the project repo level instead of the reusable app level.
4. Submit a PR to the official repo so that pip can get upgraded

That's all I can think of. Solutions 1. and 2. are really bad IMHO, 3. is even worse, and 4. can take time, which I can't really afford.

Any better ideas?

Answer 1

The pip notation is not the same as the the setuptools notation. There is no immediate reason to change the setup script. The install_requires and dependency_links seem fine in the first version, and probably actually are if it used to work.

Now, it is true that the newer version of pip do not take the dependency_links of setuptools into account at all. So these custom dependency links can be added to a requirements.txt file instead (with the pip notation of course).

Additionally you could choose to entirely remove the dependency_links argument in the setup script to avoid duplicates.

Answer 2

I don't know if it's the right answer, but I used this workaround

So basically in my setup.py I add a function and call it:

def install_github_package(github_url):
    try:
        import github_url
        # ... do some version checking ...
    except (ModuleNotFoundError, ImportError):
        if '--user' in sys.argv:
            subprocess.run([sys.executable, '-m', 'pip', 'install', '--upgrade',
                            '--user', github_url], check=False)
        else:
            subprocess.run([sys.executable, '-m', 'pip', 'install', '--upgrade',
                            github_url], check=False)

install_github_package("git+https://github.com/company/package.git")