147

I have a git repository with many folders, one of them being a python module installable with pip, like this:

repo.git/
repo.git/folder1/
repo.git/folder2/
repo.git/mymodule/
repo.git/mymodule/__init__.py
repo.git/mymodule/setup.py
repo.git/mymodule/...

Right now I have to do the following to install:

git clone http://server/repo.git
cd repo
pip install mymodule
cd ..
rm -rf repo

Is it possible to install the module directly with pip without explicitly cloning ?

I tried:

pip install git+https://server/repo.git/mymodule/
pip install git+https://server/repo.git:mymodule/

But I get:

IOError: [Errno 2] No such file or directory: '/tmp/pip-88tlLm-build/setup.py'
OJFord
  • 10,522
  • 8
  • 64
  • 98
J. Martinot-Lagarde
  • 3,280
  • 2
  • 15
  • 17
  • 1
    This question seems to already be answered: http://stackoverflow.com/questions/10847764/pip-install-from-git-repo – synthesizerpatel Nov 26 '12 at 20:47
  • 4
    The question you linked was about missing files because there's no MANIFEST.in. My question is about how to install a subdirectory with pip, no files are missing since I can install the package by cloning the whole repo. – J. Martinot-Lagarde Nov 27 '12 at 07:59
  • 1
    @synthesizerpatel The question you linked to asks about different thing but the title was misleading so I corrected it. You can consider removing your (I guess) downvote. – Piotr Dobrogost Nov 27 '12 at 21:29

2 Answers2

185

There is a pull request regarding this feature, and it seems to have been merged to develop branch a month ago. The syntax is the following:

pip install -e "git+https://git.repo/some_repo.git#egg=$NAME_OF_PACKAGE&subdirectory=$SUBDIR_IN_REPO" # install a python package from a repo subdirectory

We probably have to wait for a while until it gets merged to master and is distributed.

UPDATE: This is now available and documented at https://pip.pypa.io/en/stable/cli/pip_install/#vcs-support as follows:

For projects where setup.py is not in the root of project, "subdirectory" component is used. Value of "subdirectory" component should be a path starting from root of the project to where setup.py is located.

So if your repository layout is:

- pkg_dir/
  - setup.py  # setup.py for package ``pkg``
  - some_module.py
- other_dir/
  - some_file
- some_other_file

You'll need to use

pip install -e "vcs+protocol://repo_url/#egg=pkg&subdirectory=pkg_dir"

Note: Make sure to surround it with quotes otherwise subdirectory won't work.

Edward Z. Yang
  • 26,325
  • 16
  • 80
  • 110
Dennis Golomazov
  • 16,269
  • 5
  • 73
  • 81
  • 4
    I just checked and it is part of pip v1.5, released Jan 1 2014: https://github.com/pypa/pip/blob/develop/CHANGES.txt – tomka Feb 10 '14 at 14:10
  • 1
    [Specific changelog entry](https://github.com/pypa/pip/blob/54de36d9aee18b23379cd417dcc6ede3c0d93f96/CHANGES.txt#L57) in pip v1.5 where this feature was merged. – gene_wood May 19 '14 at 22:35
  • 1
    This is in theory what I want, but I can't seem to get it to work with git+ssh or git+https, even though I have pip version 1.5.6 working with Python 3.4. I get the error, `'subdirectory' is not recognized as an internal or external command, operable program or batch file.` – Elaine Hale Jul 22 '14 at 16:52
  • 26
    If you are using ubuntu make sure to use single quotation marks before the actual git location, otherwise the &subdirectory is ignored! For example: pip install -e 'git+https://git.repo/some_repo.git#egg=version_subpkg&subdirectory=version_subpkg' – Lin Nov 09 '14 at 12:21
  • 12
    Just a heads-up: this does not seem to work (at least on pip v1.5.6, not sure about 6+ versions) for packages whose *entry points* lie within the subdirectory. `pip install -e 'git+https://…/repo.git@branch#egg=package&subdirectory=package'` results in the package *appearing* to install, and a `package.egg-link` file getting put in site-packages — but it points to the root directory of `src/repo`, *not* to `src/repo/package`, as I would expect. This means that `py_modules` can’t be imported, and `entry_points` don’t actually work: they both raise an `ImportError: No module named package`. – Mark G. Dec 07 '15 at 17:57
  • 2
    `-e` seems necessary, otherwise `pip freeze` doesn't show the whole path, only the egg. This happened on `pip-1.5.4` – zyxue Mar 26 '16 at 15:29
  • 2
    tip for others: you HAVE to use git+https, git+git will not work, you'll get fatal: remote error: Repository not found. presumably because it doesn't parse the "query" string correctly. I've just logged: https://github.com/pypa/pip/issues/4898 – amohr Nov 30 '17 at 22:45
  • 1
    Another tip for others: On Windows, you must place the URL in double quotes, or you'll get an error "'subdirectory' is not recognized as an internal or external command". E.g., use: pip install -e "git+myurl/proj#egg=subpkg&subdirectory=subdir" –  Dec 04 '18 at 16:04
  • @JohnPirie thanks! Feel free to update the answer accordingly. – Dennis Golomazov Dec 04 '18 at 17:44
  • Updated link to documentation for PIP: https://pip.pypa.io/en/stable/topics/vcs-support/#url-fragments – Brendano257 Feb 13 '23 at 12:06
48

It's been already stated in one of the comments under the correct answer, but just to highlight this issue: when executing this from Linux command line, you must escape the &-character since ampersand is telling the command line to run a command in background:

git+https://git.repo/some_repo.git#egg=version_subpkg\&subdirectory=repo

Notice the backslash before the ampersand. The escaping behaviour might depend on the Linux distro; I'm not an expert.
If you ignore this, you might run into a cryptic error like the following:

bash: (...) command not found
Michael
  • 8,362
  • 6
  • 61
  • 88
nichoio
  • 6,289
  • 4
  • 26
  • 33
  • 3
    Would enclosing the url in single quotes help avoid the need for escaping the ampersand? I know that is helpful when working with URLs in cURL. – Josh Peak Oct 22 '18 at 21:43
  • 1
    Also worth noting: When executing this from Windows command line, you must place the URL in double quotes, or you'll get an error "'subdirectory' is not recognized as an internal or external command". E.g., use pip install -e "git+https://myurl/proj#egg=subpkg&subdirectory=subdir" –  Dec 04 '18 at 16:02
  • In general, when working with shell scripts, use double quotes around arguments to be on the safe side. for more best practices see: https://www.shellcheck.net/ or https://github.com/koalaman/shellcheck – Erik Aronesty Jan 17 '19 at 15:38
  • 2
    The escaping is necessary in OSX as well (as it is a sort of Linux distro) – ygesher Mar 03 '19 at 16:53
  • 13
    For branch-based install use: `pip install git+ssh://git@github.com/org_or_username/repo.git@branch#subdirectory=path/to/dubdir` – RDK Sep 06 '19 at 06:41