544

I am trying to install a Python package from a private GitHub repository. For a public repository, I can issue the following command which works fine:

pip install git+git://github.com/django/django.git

However, if I try this for a private repository:

pip install git+git://github.com/echweb/echweb-utils.git

I get the following output:

Downloading/unpacking git+git://github.com/echweb/echweb-utils.git
Cloning Git repository git://github.com/echweb/echweb-utils.git to /var/folders/cB/cB85g9P7HM4jcPn7nrvWRU+++TI/-Tmp-/pip-VRsIoo-build
Complete output from command /usr/local/bin/git clone git://github.com/echweb/echweb-utils.git /var/folders/cB/cB85g9P7HM4jcPn7nrvWRU+++TI/-Tmp-/pip-VRsIoo-build:
fatal: The remote end hung up unexpectedly

Cloning into /var/folders/cB/cB85g9P7HM4jcPn7nrvWRU+++TI/-Tmp-/pip-VRsIoo-build...

----------------------------------------
Command /usr/local/bin/git clone git://github.com/echweb/echweb-utils.git /var/folders/cB/cB85g9P7HM4jcPn7nrvWRU+++TI/-Tmp-/pip-VRsIoo-build failed with error code 128

I guess this is because I am trying to access a private repository without providing any authentication. I therefore tried to use Git + ssh hoping that pip would use my SSH public key to authenticate:

pip install git+ssh://github.com/echweb/echweb-utils.git

This gives the following output:

Downloading/unpacking git+ssh://github.com/echweb/echweb-utils.git
Cloning Git repository ssh://github.com/echweb/echweb-utils.git to /var/folders/cB/cB85g9P7HM4jcPn7nrvWRU+++TI/-Tmp-/pip-DQB8s4-build
Complete output from command /usr/local/bin/git clone ssh://github.com/echweb/echweb-utils.git /var/folders/cB/cB85g9P7HM4jcPn7nrvWRU+++TI/-Tmp-/pip-DQB8s4-build:
Cloning into /var/folders/cB/cB85g9P7HM4jcPn7nrvWRU+++TI/-Tmp-/pip-DQB8s4-build...

Permission denied (publickey).

fatal: The remote end hung up unexpectedly

----------------------------------------
Command /usr/local/bin/git clone ssh://github.com/echweb/echweb-utils.git /var/folders/cB/cB85g9P7HM4jcPn7nrvWRU+++TI/-Tmp-/pip-DQB8s4-build failed with error code 128

Is what I am trying to achieve even possible? If so, how can I do it?

Peter Mortensen
  • 30,738
  • 21
  • 105
  • 131
Adam J. Forster
  • 17,789
  • 9
  • 25
  • 20
  • 4
    It certainly isn't the correct answer, but cloning the repo manually then pointing pip to localhost instead of github will get you past this if you're just trying to be productive. – nmichaels Jan 28 '11 at 16:57
  • 2
    @nmicheals That's what I've done so far, but I need to put this into requirements files for deployment across many web sites all with separate virtualenvs. – Adam J. Forster Jan 28 '11 at 18:04
  • Just to he sure: you've already set up ssh key support on github, right? If that's definitely not working.... Have you tried using git+git://user:pass@github.com/.... as the URI? – Jason LeBrun Jan 29 '11 at 02:17
  • Try using `eval $(ssh-agent); ssh-add ~/.ssh/github_id_rsa` and then run pip installs following https://pip.pypa.io/en/stable/reference/pip_install/#git – Darren Weber Feb 16 '19 at 00:44

17 Answers17

585

You can use the git+ssh URI scheme, but you must set a username. Notice the git@ part in the URI:

pip install git+ssh://git@github.com/echweb/echweb-utils.git

Also read about deploy keys.

PS: In my installation, the "git+ssh" URI scheme works only with "editable" requirements:

pip install -e URI#egg=EggName

Remember: Change the : character that git remote -v prints to a / character before using the remote's address in the pip command:

$ git remote -v
origin  git@github.com:echweb/echweb-utils.git (fetch)
#                     ^ change this to a '/' character

If you forget, you will get this error:

ssh: Could not resolve hostname github.com:echweb:
         nodename nor servname provided, or not known
Michael
  • 8,362
  • 6
  • 61
  • 88
oxyum
  • 6,387
  • 1
  • 15
  • 13
  • 7
    Thanks for that, I was just missing the git@ part. Incidentally the command 'pip install git+ssh://git@github.com/echweb/echweb-utils.git' worked, I did not require the -e switch. – Adam J. Forster Jan 31 '11 at 09:02
  • 4
    you can also use an .ssh/config file to set the proper username – Stefano Apr 12 '13 at 11:33
  • 3
    This used to be working for me but now isn't, I'm even using the correct `git+ssh://git@github.com/echweb/echweb-utils.git` scheme in my requirements file. I opened a new question here: http://stackoverflow.com/questions/18883430/trouble-installing-private-github-repository-using-pip. Any help would be great. – Robeezy Sep 18 '13 at 22:34
  • This is what I need to do with our in-house (corporate) github deployment. Is the "git@" part documented anywhere? I spent some time looking before I found your post; it definitely doesn't seem to be a part of pip documentation. I'm curious. And thank you! – m.kocikowski Mar 04 '14 at 23:13
  • It's not pip-specific, so it not part of pip doc. I know that because have strong knowledge in unix/ssh/git and setup repo hostings in pre-github(and pre-git) era. "git@" part can be found in github "SSH clone URL" field for any repo in which you have write access. – oxyum Mar 05 '14 at 11:09
  • @m.kocikowski "git@" is part of the URL. URLs are standard, and not pip-specific: https://tools.ietf.org/html/rfc3986 – WhyNotHugo Jul 26 '14 at 07:16
  • Does it matter what username you use? – AnnanFay Nov 04 '14 at 14:26
  • Yes, you should use only "git" username for ssh access to github and most other VCS hostings. All files saved under one system user (git) and permissions managed not by OS, but by software internally. To distinguish between users special ssh-daemon uses key, which was used for auth. – oxyum Nov 05 '14 at 00:01
  • 3
    Perfect. Putting `git+ssh://git@github.com/echweb/echweb-utils.git` into `requirements.txt` works too, which is awesome. – Avindra Goolcharan Feb 29 '16 at 13:53
  • 4
    If you need to run this from a script (eg: for deployment...), you can specify which SSH key to use with the environment variable `GIT_SSH_COMMAND="ssh -i ~/.ssh/my-deploy-key"` – Laurent S Dec 20 '16 at 19:12
  • "Could not detect requirement name, please specify one with #egg=" – User Jan 16 '17 at 00:47
  • 2
    If you want to install from a specific branch: `pip install git+ssh://git@github.com/echweb/echweb-utils.git@branch-name` – Guilherme Beltramini Jun 27 '17 at 13:46
  • What is the purpose of giving #egg=EggName ? – Garry Mccarthy Feb 23 '22 at 15:32
  • Hmmm, not sure if it's still actual. Before `wheels`, there were `eggs` in the world of Python, but as of now, it's most likely totally outdated. – oxyum Feb 24 '22 at 16:05
  • thank you! i wasn't adding the `git@` and it was using the wrong username. – greenie-beans Jan 10 '23 at 20:21
100

As an additional technique, if you have the private repository cloned locally, you can do:

pip install git+file://c:/repo/directory

More modernly, you can just do this (and the -e will mean you don't have to commit changes before they're reflected):

pip install -e C:\repo\directory
Peter Mortensen
  • 30,738
  • 21
  • 105
  • 131
Scott Stafford
  • 43,764
  • 28
  • 129
  • 177
54

You can do it directly with the HTTPS URL like this:

pip install git+https://github.com/username/repo.git

This also works just appending that line in the requirements.txt in a Django project, for instance.

Peter Mortensen
  • 30,738
  • 21
  • 105
  • 131
danius
  • 2,664
  • 27
  • 33
  • 65
    For private repos, though, this will trigger a username/password prompt on the console, which is probably not what you want. – rascalking Jul 27 '17 at 14:56
43

It also works with Bitbucket:

pip install git+ssh://git@bitbucket.org/username/projectname.git

Pip will use your SSH keys in this case.

Peter Mortensen
  • 30,738
  • 21
  • 105
  • 131
gnrfan
  • 19,071
  • 1
  • 21
  • 12
34

I found it much easier to use tokens than SSH keys. I couldn't find much good documentation on this, so I came across this solution mainly through trial and error. Further, installing from pip and setuptools have some subtle differences; but this way should work for both.

GitHub don't (currently, as of August 2016) offer an easy way to get the zip / tarball of private repositories. So you need to tell setuptools that you're pointing to a Git repository:

from setuptools import setup
import os
# Get the deploy key from https://help.github.com/articles/git-automation-with-oauth-tokens/
github_token = os.environ['GITHUB_TOKEN']

setup(
    # ...
    install_requires='package',
    dependency_links = [
    'git+https://{github_token}@github.com/user/{package}.git/@{version}#egg={package}-0'
        .format(github_token=github_token, package=package, version=master)
        ]

A couple of notes here:

  • For private repositories, you need to authenticate with GitHub; the simplest way I found is to create an OAuth token, drop that into your environment, and then include it with the URL
  • You need to include some version number (here is 0) at the end of the link, even if there's isn't any package on PyPI. This has to be a actual number, not a word.
  • You need to preface with git+ to tell setuptools it's to clone the repository, rather than pointing at a zip / tarball
  • version can be a branch, a tag, or a commit hash
  • You need to supply --process-dependency-links if installing from pip
Maximilian
  • 7,512
  • 3
  • 50
  • 63
28

I figured out a way to automagically 'pip install' a GitLab private repository that requires no password prompt. This approach uses GitLab "Deploy Keys" and an SSH configuration file, so you can deploy using keys other than your personal SSH keys (in my case, for use by a 'bot). Perhaps someone kind soul can verify using GitHub.

Create a New SSH key:

ssh-keygen -t rsa -C "GitLab_Robot_Deploy_Key"

The file should show up as ~/.ssh/GitLab_Robot_Deploy_Key and ~/.ssh/GitLab_Robot_Deploy_Key.pub.

Copy and paste the contents of the ~/.ssh/GitLab_Robot_Deploy_Key.pub file into the GitLab "Deploy Keys" dialog.

Test the New Deploy Key

The following command tells SSH to use your new deploy key to set up the connection. On success, you should get the message: "Welcome to GitLab, UserName!"

ssh -T -i ~/.ssh/GitLab_Robot_Deploy_Key git@gitlab.mycorp.com

Create the SSH Configuration File

Next, use an editor to create a ~/.ssh/config file. Add the following contents. The 'Host' value can be anything you want (just remember it, because you'll be using it later). The HostName is the URL to your GitLab instance. The IdentifyFile is path to the SSH key file you created in the first step.

Host GitLab
  HostName gitlab.mycorp.com
  IdentityFile ~/.ssh/GitLab_Robot_Deploy_Key

Point SSH to the Configuration file

oxyum gave us the recipe for using pip with SSH:

pip install git+ssh://git@gitlab.mycorp.com/my_name/my_repo.git

We just need to modify it a bit to make SSH use our new Deploy Key. We do that by pointing SSH to the Host entry in the SSH configuration file. Just replace the 'gitlab.mycorp.com' in the command to the host name we used in the SSH configuration file:

pip install git+ssh://git@GitLab/my_name/my_repo.git

The package should now install without any password prompt.

Reference A
Reference B

Peter Mortensen
  • 30,738
  • 21
  • 105
  • 131
JS.
  • 14,781
  • 13
  • 63
  • 75
21

If you want to install dependencies from a requirements file within a CI server, you can do this:

git config --global credential.helper 'cache'
echo "protocol=https
host=example.com
username=${GIT_USER}
password=${GIT_PASS}
" | git credential approve
pip install -r requirements.txt

In my case, I used GIT_USER=gitlab-ci-token and GIT_PASS=${CI_JOB_TOKEN}.

This method has a clear advantage. You have a single requirements file which contains all of your dependencies.

AmirHossein
  • 1,310
  • 1
  • 12
  • 19
  • I am wondering if I can do the same in Github , like github has GITHUB_TOKEN as default token attached to each repo just like in gitlab CI_JOB_TOKEN but I am not sure of user in Github ? – vgdub Dec 07 '21 at 15:16
  • Providing username+password the first time, and doing one time `git config --global credential.helper 'store'` solved my problem. – Dženan Jun 21 '23 at 14:01
18

The syntax for the requirements file is given here:

https://pip.pypa.io/en/latest/reference/pip_install.html#requirements-file-format

So for example, use:

-e git+http://github.com/rwillmer/django-behave#egg=django-behave

if you want the source to stick around after installation.

Or just

git+http://github.com/rwillmer/django-behave#egg=django-behave

if you just want it to be installed.

Peter Mortensen
  • 30,738
  • 21
  • 105
  • 131
Rachel
  • 2,858
  • 2
  • 26
  • 30
  • The URL for the requirements file syntax has changed. The new URL is: https://pip.pypa.io/en/latest/reference/pip_install.html#requirements-file-format – sphere May 22 '15 at 09:37
  • Is the `egg` a hard requirement? I'm trying to install via ssh a private python package with a deploy key but keep getting a response of `ERROR: Command errored out with exit status 128`. – Alex F Oct 13 '20 at 13:42
18

If you need to do this in, say, a command line one-liner, it's also possible. I was able to do this for deployment on Google Colab:

  1. Create a Personal Access Token: https://docs.github.com/en/github/authenticating-to-github/creating-a-personal-access-token
  2. Run: pip install git+https://<USERNAME>:<PERSONAL ACCESS TOKEN>@github.com/<ACCOUNT>/<REPOSITORY>.git
Community
  • 1
  • 1
Jamie
  • 2,181
  • 1
  • 21
  • 35
  • 1
    You might add a comment noting that your access token will be in plaintext and accessible to anyone who can read the pip configuration. – Michael Aug 18 '20 at 14:22
  • Also, now that I think about it, your answer is kind of a duplicate of [this one](https://stackoverflow.com/a/39254921/241211) from 4 years ago. – Michael Aug 18 '20 at 14:29
16

My case was kind of more complicated than most of the ones described in the answers. I was the owner of two private repositories repo_A and repo_B in a Github organization and needed to pip install repo_A during the python unittests of repo_B, as a Github action.

Steps I followed to solve this task:

  • Created a Personal Access Token for my account. As for its permissions, I only needed to keep the default ones, .i.e. repo - Full control of private repositories.
  • Created a repository secret under repo_B, pasted my Personal Access Token in there and named it PERSONAL_ACCESS_TOKEN. This was important because, unlike the solution proposed by Jamie, I didn't need to explicitly expose my precious raw Personal Access Token inside the github action .yml file.
  • Finally, pip install the package from source via HTTPS (not SSH) as follows:
export PERSONAL_ACCESS_TOKEN=${{ secrets.PERSONAL_ACCESS_TOKEN }}

pip install git+https://${PERSONAL_ACCESS_TOKEN}@github.com/MY_ORG_NAME/repo_A.git
pcko1
  • 833
  • 12
  • 22
  • 3
    Would SSH keys have worked in this case as well? – Marcel Wilson Aug 12 '21 at 16:30
  • 1
    @MarcelWilson I would expect it to work. However, using an access token over SSH keys comes with the additional benefit of being able to adjust the access level of the token. This might be very useful when using Github `actions` (workflows) to `pip install` a private repo during unittesting of another shared repo where you don't want to share your SSH keys with others, not even as a Github`secret`. Tl;dr, access tokens are less permissive as far as I can tell. – pcko1 Aug 12 '21 at 17:26
  • @MarcelWilson - what pcko1 said, for me, the primary plus of token is that I can create and destroy tokens as needed without mucking with my personal dev settings – meyerson Dec 21 '21 at 15:45
  • To use ssh, one would have to use git+ssh instead of git+https though no need to provide a personal access token in this case. – Thomas BDX May 26 '23 at 13:03
10

You can also install a private repository dependency via git+https://github.com/... URL by providing login credentials (login and password, or deploy token) for curl with the .netrc file:

echo "machine github.com login ei-grad password mypasswordshouldbehere" > ~/.netrc
pip install "git+https://github.com/ei-grad/my_private_repo.git#egg=my_private_repo"
Peter Mortensen
  • 30,738
  • 21
  • 105
  • 131
ei-grad
  • 792
  • 7
  • 19
8

If you don't want to use SSH, you could add the username and password in the HTTPS URL.

The code below assumes that you have a file called "pass" in the working directory that contains your password.

export PASS=$(cat pass)
pip install git+https://<username>:$PASS@github.com/echweb/echweb-utils.git
Peter Mortensen
  • 30,738
  • 21
  • 105
  • 131
sam
  • 1,243
  • 12
  • 10
7

When I was installing from GitHub I was able to use:

pip install git+ssh://git@github.com/<username>/<projectname>.git#egg=<eggname>

But, since I had to run pip as sudo, the SSH keys were not working with GitHub any more, and "git clone" failed on "Permission denied (publickey)". Using git+https allowed me to run the command as sudo, and have GitHub ask me for my user/password.

sudo pip install git+https://github.com/<username>/<projectname>.git#egg=<eggname>
Peter Mortensen
  • 30,738
  • 21
  • 105
  • 131
jcarballo
  • 27,395
  • 3
  • 28
  • 28
  • Do other user accounts need to access your project? If not, have you discovered [nerdwaller's suggestion?](https://stackoverflow.com/questions/4830856/#comment55105837_27901245) – Michael Apr 06 '17 at 15:48
  • 1
    Is the `egg` a hard requirement? I'm trying to install via ssh a private python package with a deploy key but keep getting a response of `ERROR: Command errored out with exit status 128`. – Alex F Oct 13 '20 at 13:43
4

If you have your own library/package on GitHub, GitLab, etc., you have to add a tag to commit with a concrete version of the library, for example, v2.0, and then you can install your package:

pip install git+ssh://link/name/repo.git@v2.0

This works for me. Other solutions haven't worked for me.

Peter Mortensen
  • 30,738
  • 21
  • 105
  • 131
zolty13
  • 1,943
  • 3
  • 17
  • 34
3

Just copy the remote from the original git clone command (or from git remote -v). You will get something like this:

  • Bitbucket: git+ssh://git@bitbucket.org:your_account/my_pro.git

  • GitHub: git+ssh://git@github.com:your_account/my_pro.git

Next, you need to replace : with / next to the domain name.

So install using:

pip install git+ssh://git@bitbucket.org/your_account/my_pro.git
Michael
  • 8,362
  • 6
  • 61
  • 88
Mohideen bin Mohammed
  • 18,813
  • 10
  • 112
  • 118
  • What do you mean by *"copy git clone command"*? Do you mean *"copy using the 'git clone' command"*? Or *"copy the 'git clone' command"*? Or something else? (Please respond by [editing your answer](https://stackoverflow.com/posts/58689622/edit), not here in comments (as appropriate). Thanks in advance.). – Peter Mortensen Dec 11 '19 at 22:38
  • 2
    if you visit the repository website and press the clone button on top both BB and github will show you a clone command for the repo. – florian Jul 28 '20 at 20:09
0

oxyum's solution is OK for this answer. I just want to point out that you need to be careful if you are installing using sudo as the keys must be stored for root too (for example, /root/.ssh).

Then you can type

sudo pip install git+ssh://git@github.com/echweb/echweb-utils.git
Peter Mortensen
  • 30,738
  • 21
  • 105
  • 131
Andres Romero
  • 69
  • 3
-1

You may try 

pip install git+git@gitlab.mycorp.com/my_name/my_repo.git

without ssh:.... That works for me.

Peter Mortensen
  • 30,738
  • 21
  • 105
  • 131
user2235670
  • 11
  • 1