Configuring so that pip install can work from github

Question

We'd like to use pip with github to install private packages to our production servers. This question concerns what needs to be in the github repo in order for the install to be successful.

Assuming the following command line (which authenticates just fine and tries to install):

pip install git+ssh://git@github.com/BlahCo/search/tree/prod_release_branch/ProductName

What needs to reside in the ProductName? Is it the contents of what would normally be in the tar file after running setup.py with the sdist option, or is the actual tar.gz file, or something else?

I'm asking here because I've tried several variations and can't make it work. Any help appreciated.

score 332 · Accepted Answer · edited Jul 17 '23 at 01:28

332

You need the whole python package, with a setup.py file in it.

A package named foo would be:

foo # the installable package
├── foo
│   ├── __init__.py
│   └── bar.py
└── setup.py

And install from github like:

$ pip install git+ssh://git@github.com/myuser/foo.git
or
$ pip install git+https://github.com/myuser/foo.git@v123
or
$ pip install git+https://github.com/myuser/foo.git@newbranch

More info at https://pip.pypa.io/en/stable/cli/pip_install/

edited Jul 17 '23 at 01:28

The Empty String Photographer

127
12

answered Nov 24 '11 at 11:40

Hugo Lopes Tavares

28,528
5
47
45

4

The above works, thank you very much. But what if I have releases in a subdir within a repo, so rather than foo.git I'm looking for foo/releases/ProductVer . Is that possible and if so how? Thanks very much for the help! – ccgillett Nov 25 '11 at 17:13
4

No, it is not possible. pip only installs from root repository directory, at least for git. Don't know how subversion behaves... – Hugo Lopes Tavares Nov 25 '11 at 18:00
3

If you want to do this over ssh and private repos, [this is a post on how to do that](http://jtushman.github.io/blog/2013/06/17/sharing-code-across-applications-with-python/) – Jonathan Jun 17 '13 at 18:34
Why is it that you did not pass the `-e` option (editable mode) to pip? – Amelio Vazquez-Reina Jul 08 '13 at 22:54
@user815423426 because I don't want it to be an `editable` installation. For more details take a look at http://pythonhosted.org/setuptools/setuptools.html#development-mode – Hugo Lopes Tavares Jul 09 '13 at 17:30
14

Here's the new url scheme: `pip install git+https://github.com/pypa/pip.git` Source: [pip Github repo](https://github.com/pypa/pip/issues/539#issuecomment-5825986) – aboutaaron Aug 06 '13 at 05:00
@ccgillett It's a long time later, but there is a way to specify a subfolder: https://pip.pypa.io/en/stable/topics/vcs-support/#url-fragments – daviewales Jan 19 '23 at 04:52

score 120 · Answer 2 · edited Jul 28 '15 at 14:52

120

I had similar issue when I had to install from github repo, but did not want to install git , etc.

The simple way to do it is using zip archive of the package. Add /zipball/master to the repo URL:

    $ pip install https://github.com/hmarr/django-debug-toolbar-mongo/zipball/master
Downloading/unpacking https://github.com/hmarr/django-debug-toolbar-mongo/zipball/master
  Downloading master
  Running setup.py egg_info for package from https://github.com/hmarr/django-debug-toolbar-mongo/zipball/master
Installing collected packages: django-debug-toolbar-mongo
  Running setup.py install for django-debug-toolbar-mongo
Successfully installed django-debug-toolbar-mongo
Cleaning up...

This way you will make pip work with github source repositories.

edited Jul 28 '15 at 14:52

Alastair McCormack

26,573
8
77
100

answered Dec 05 '11 at 08:48

Dmitry

1,246
1
7
4

3

This is the only answer that also works under Windows – divenex Mar 30 '17 at 08:31
1

How can this be done with private repositories? (as the question asks) – Rodrigo E. Principe Feb 19 '20 at 16:43

score 35 · Answer 3 · edited Aug 02 '22 at 19:51

35

If you want to use requirements.txt file, you will need git and something like the entry below to anonymously fetch the master branch in your requirements.txt.

For regular install:

git+git://github.com/celery/django-celery.git

For "editable" install:

-e git://github.com/celery/django-celery.git#egg=django-celery

Editable mode downloads the project's source code into ./src in the current directory. It allows pip freeze to output the correct github location of the package.

edited Aug 02 '22 at 19:51

Huessy

111
8

answered Aug 18 '16 at 11:14

wieczorek1990

7,403
1
25
19

3

where does that egg names comes from? it fails to load package in django after replacing pip package with github repo – holms Mar 07 '18 at 16:01
@holms Lookup name in setup.py – wieczorek1990 Jun 24 '19 at 09:14
What's `#egg=django-celery`? Is that a placeholder? – enchance Feb 15 '20 at 17:34
@enchance It's the package name. – wieczorek1990 Mar 16 '20 at 09:51
1

I found `git+git://github.com` wouldn't work on Windows (pip had a failure when cloning) but `git+https://github.com` worked. – idbrii Feb 04 '21 at 19:02

score 16 · Answer 4 · answered Sep 01 '16 at 09:16

16

Clone target repository same way like you cloning any other project:

git clone git@github.com:myuser/foo.git

Then install it in develop mode:

cd foo
pip install -e .

You can change anything you wan't and every code using foo package will use modified code.

There 2 benefits ot this solution:

You can install package in your home projects directory.
Package includes .git dir, so it's regular Git repository. You can push to your fork right away.

answered Sep 01 '16 at 09:16

avalanchy

814
1
11
19

3

I can testify that this solution is magical. In my case, I wanted to hack on `pip`, so I cloned the `pip` repository, created a virtualenv, activated it, and did `pip install -e .`. Then the `pip` in the virtualenv was in development mode! I'm impressed that this works even with the package manager itself. – Resigned June 2023 May 18 '17 at 23:30
This is great! I notice that when I installed using this method and then run `pip list`, the package in question has references to the Git branch and absolute path from which it was installed. Does it keep any references to those or can the source be deleted? – MadPhysicist Aug 18 '17 at 16:22

score 5 · Answer 5 · answered Aug 19 '21 at 18:52

Here is the simple solution

With git

pip install git+https://github.com/jkbr/httpie.git

Without git

pip install https://github.com/jkbr/httpie/tarball/master

or

pip install https://github.com/jkbr/httpie/zipball/master

or

pip install https://github.com/jkbr/httpie/archive/master.zip

Note: You need a python package with the setup.py file in it.

score 2 · Answer 6 · answered Dec 07 '21 at 06:21

2

Below format could be use to install python libraries via pip from GitHub.

pip install <LibName>@git+ssh://git@github.com/<username>/<LibName>#egg<LibName>

answered Dec 07 '21 at 06:21

Abhishake Gupta

2,939
1
25
34

score 1 · Answer 7 · answered May 08 '20 at 08:50

1

you can try this way in Colab

!git clone https://github.com/UKPLab/sentence-transformers.git
!pip install -e /content/sentence-transformers
import sentence_transformers

answered May 08 '20 at 08:50

Shaina Raza

1,474
17
12

score 1 · Answer 8 · answered Jun 16 '22 at 13:18

Tested Optimized Ubuntu Solution using the terminal command:

Step 1: In a selected directory clone the git repo.

Example:

$ git clone https://github.com/httpie/httpie.git

Step 2: select/change path to the directory, to the cloned folder

$ cd ClonedFolderName

Step 3: Enter following command to install that package

ColnedFolderName(directory Name) $ pip install ./

pip install ./ is command to enter in cloned directory name

Note: Make sure setup.py is inside cloned repo. (which is by default in it)

Configuring so that pip install can work from github

8 Answers8

For regular install:

For "editable" install:

Here is the simple solution

With git

Without git

Linked

Related