How to `pip install` a package that has Git dependencies?

Question

I have a private library called some-library (actual names have been changed) with a setup file looking somewhat like this:

setup(
    name='some-library',

    // Omitted some less important stuff here...

    install_requires=[
        'some-git-dependency',
        'another-git-dependency',
    ],
    dependency_links=[
        'git+ssh://git@github.com/my-organization/some-git-dependency.git#egg=some-git-dependency',
        'git+ssh://git@github.com/my-organization/another-git-dependency.git#egg=another-git-dependency',
    ],
)

All of these Git dependencies may be private, so installation via HTTP is not an option. I can use python setup.py install and python setup.py develop in some-library's root directory without problems.

However, installing over Git doesn't work:

pip install -vvv -e 'git+ssh://git@github.com/my-organization/some-library.git@1.4.4#egg=some-library'

The command fails when it looks for some-git-dependency, mistakenly assumes it needs to get the dependency from PyPI and then fails after concluding it's not on PyPI. My first guess was to try re-running the command with --process-dependency-links, but then this happened:

   Cannot look at git URL git+ssh://git@github.com/my-organization/some-git-dependency.git#egg=some-git-dependency
   Could not find a version that satisfies the requirement some-git-dependency (from some-library) (from versions: )

Why is it producing this vague error? What's the proper way to pip install a package with Git dependencies that might be private?

Answer 1

What's the proper way to pip install a package with Git dependencies that might be private?

Two options

1. Use dependency_links as you do. See below for details.

2. Along side the dependency_links in your setup.py's, use a special dependency-links.txt that collects all the required packages. Then add this package in requirements.txt. That's my recommendend option as explained below.

   # dependency-links.txt
   git+ssh://...@tag#egg=package-name1
   git+ssh://...@tag#egg=package-name2
   # requirements.txt (per deployed application)
   -r dependency-links.txt
   

While option 2 adds some extra burden on package management, namely keeping dependency-links.txt up to date, it makes installing packages a lot easier because you can' forget to add the --process-dependency-link option on pip install.

Perhaps more importantly, using dependency-links.txt you get to specify the exact version to be installed on deployment, which is want you want in a CI/CD environment - nothing is more risky than to install some version. From a package maintainer's perspective however it is common and considered good practice to specify a minimum version, such as

    # setup.py in a package
    ...
       install_requires = [ 'foo>1.0', ... ]

That's great because it makes your packages work nicely with other packages that have similar dependencies yet possibly on different versions. However, in a deployed application this can still cause mayhem if there are conflicting requirements among packages. E.g. package A is ok with foo>1.0, package B wants foo<=1.5 and the most recent version is foo==2.0. Using dependency-links.txt you can be specific, applying one version for all packages:

    # dependency-links.txt
    foo==1.5

The command fails when it looks for some-git-dependency,

To make it work, you need to add --process-dependency-links for pip to recognize the dependency to github, e.g.

pip install --process-dependency-links -r private-requirements.txt

Note since pip 8.1.0 you can add this option to requirements.txt. On the downside it gets applied to all packages installed and may have unintended consequences. That said, I find using dependency-links.txt is a safer and more manageable solution.

All of these Git dependencies may be private

There are three options:

1. Add collaborators on each of the required packages' repositories. These collaborators need to have their ssh keys setup with github for this to work. Then use git+ssh://...

2. Add a deploy key to each of the repositories. The downside here is that you need to distribute the corresponding private key to all the machines that need to deploy. Again use git+ssh://...

3. Add a personal access token on the github account that holds the private repositories. Then you can use git+https://accesstoken@github.com/... The downside is that the access token will have read + write access to all repositories, public and private, on the respective github account. On the plus side distributing and managing per-repository private keys is no longer necessary, and cycling the key is a lot simpler. In an all-inhouse environment where every dev has access to all repositories I have found this to be the most efficient, hassle-free way for everybody. YMMV

Answer 2

This should work for private repositories as well:

dependency_links = [
     'git+ssh://git@github.com/my-organization/some-git-dependency.git@master#egg=some-git-dependency',
     'git+ssh://git@github.com/my-organization/another-git-dependency.git@master#egg=another-git-dependency'
],

Answer 3

You should use git+git when url with #egg, like this:

-e git+git@repo.some.la:foo/my-repo.git#egg=my-repo

Use git+ssh in production without #egg, but you can specify @version or branch @master

git+ssh://git@repo.some.la/foo/my-repo.git@1.1.6

for work with app versions use git tagging Git Basics - Tagging

Answer 4

If I refer to "pip install dependency links", you would not refer to the GitHub repo itself, but to the tarball image associated to that GitHub repo:

dependency_links=[
        'git+ssh://git@github.com/my-organization/some-git-dependency/tarball/master/#egg=some-git-dependency',
        'git+ssh://git@github.com/my-organization/another-git-dependency/tarball/master/#egg=another-git-dependency',
    ],

with "some-git-dependency" being the name *and version of the dependency.

Answer 5

"Cannot look at git URL git+ssh://git@github.com/my-organization/some-git-dependency.git#egg=some-git-dependency" means pip cannot fetch an html page from this url to look for direct download links in the page, i.e, pip doesn't recognize the URL as a vcs checkout, because maybe some discrepancy between the requirement specifier and the fragment part in the vcs url.

In the case of a VCS checkout, you should also append #egg=project-version in order to identify for what package that checkout should be used.

Be sure to escape any dashes in the name or version by replacing them with underscores.

Check Dependencies that aren’t in PyPI

replace - with an _ in the package and version string.

git+ssh://git@github.com/my-organization/some-git-dependency.git#egg=some_git_dependency  

and --allow-all-external may be useful.