18

I am trying to run:

pip3 install -e .

in my Python project where I have the following setup.py:

from setuptools import setup

setup(
    name='mypackage',
    install_requires=[
        "anotherpackage@git+git@bitbucket.org:myorg/anotherpackage.git"
    ]
)

but it fails with:

error in mypackage setup command: 'install_requires' must be a string or list of strings containing valid project/version requirement specifiers; Invalid URL given

I guess it is correct about the format of my URL as PEP 508 doesn't allow specifying git user name for ssh clone URLs.

What is the correct syntax for PEP 508 URLs with git+ssh protocol for install_requires dependency for private git repositories (in this case hosted on BitBucket)? What is the syntax for specifying a specific branch, tag or sha?

More context to avoid XY problem

I have an internal Python project that depends on multiple internally developed Python packages. I would like to avoid the necessity for hosting my own PIP repository in the organisation and thus I am trying to use git URLs directly. I need to use ssh protocol for git URLs as all the users have their ssh keys configured and it would be cumbersome to ask all the users to configure their app passwords in BitBuckets (I have 2FA required and the regular user password doesn't work).

I have already tried to use:

dependency_links

setup(
    name='mypackage',
    install_requires=[
        "anotherpackage==0.0.1"
    ],
    dependency_links=[
        "git+git@bitbucket.org:myorg/anotherpackage.git@0.0.1#egg=anotherpackage-0.0.1"
    ]
)

But they are deprecated and they are ignored by pip3 install -e .. According to documentation I've found, PEP 508 URLs should be used instead.

requirements.txt file with entries duplicated from install_requires entries

I have a requirements.txt file with:

-e git+git@bitbucket.org:myorg/anotherpackage.git@0.0.1#egg=anotherpackage

and I use pip3 install -r requirements.txt instead of pip3 install -e .. It works but is suboptimal as I have to keep both setyp.py and requirements.txt in sync.

If there is any other recommended solution for my problem I would like to learn about it :)

Piotrek Bzdyl
  • 12,965
  • 1
  • 31
  • 49
  • 1
    I'm not super familiar with Python packaging, but I'm pretty familiar with Git. Are you entirely sure about the `git+git@bitbucket.org` part of your URL? I'd expect that `@` to be `://`, expressing a protocol: `git+git://bitbucket.org/...` (I'm also confused by the `anotherpackage@` prefix, but for now I'm assuming that comes from PEP 508, even though I can't find a clear explanation in the PEP itself with a quick glance.) – ChrisGPT was on strike Mar 27 '19 at 23:01
  • Thanks, yes, I know it should be `://` according to PEP508 and I tried that but it was also failing. Anyway, I delved into pip source code and figured it out. Let me add an answer. – Piotrek Bzdyl Mar 28 '19 at 08:55

1 Answers1

23

After checking pip source code I found the correct syntax for private BitBucket repositories.

The general form for the packages with URLs is <package name>@<URI> and the URI must start with a <scheme>://.

So I fixed it to:

anotherpackage@git+ssh://git@bitbucket.org:myorg/anotherpackage.git

and then I was getting a different error - this time git command (invoked by pip) was complaining about repository URL ssh://git@bitbucket.org:myorg/anotherpackage.git.

I checked the git documentation for the ssh:// URLs format and found out that hostname and organisation parts must be separated with / instead of ::

ssh://git@bitbucket.org/myorg/anotherpackage.git

This URL works fine. I also learned from the pip source code that the actual revision/branch/tag can be specified by appending @<rev-spec> so I can specify for example the tag 0.0.1 with the following in install_requires:

anotherpackage@git+ssh://git@bitbucket.org:myorg/anotherpackage.git@0.0.1

The only issue that I still have is that when I change the revision and run pip3 install -e . again it doesn't detect the change (even when run with --upgrade). I have to manually uninstall the package (pip3 uninstall anotherpackage) and run pip3 install -e . again.

Piotrek Bzdyl
  • 12,965
  • 1
  • 31
  • 49
  • 1
    Great, the only question/answer I've found for this. I think it makes sense that it won't update the dependencies even with the `--upgrade` flag, as this is the general behavior with other packages. You can always just upgrade the dependency itself if its the only thing that's changed. – rwolst Apr 04 '19 at 18:55
  • 2
    @piotrek-bzdyl, thanks for this. I thought this worked, but this ended up getting the package from pypi or cache, not fetching it from github: `Collecting mkdocs-git-committers-plugin@ git+ssh://git@github.com:ojacques/mkdocs-git-committers-plugin.git (from mkdocs-dxc==0.2.0)` `Downloading https://files.pythonhosted.org/packages/45/59/28729f789f2e74666277e40f953e22294bc3076ebf53137b93af65592874/mkdocs-git-committers-plugin-0.1.8.tar.gz` Do you experience the same? – ojacques Nov 21 '19 at 17:03
  • I have the same problem, I can't install anything from a repo. Pip is always fetching from PyPi – user1770096 Nov 22 '19 at 17:45
  • @ojacques I was using these git URLs only for my own private packages, I am not sure you can override the public ones published in pip repository. – Piotrek Bzdyl Dec 18 '19 at 07:56
  • 1
    can I use file://./mypackage ? – kerolos Oct 10 '20 at 13:43
  • Thanks for this answer @PiotrekBzdyl! The tip with the separator was what made it finally work. Perhaps you can highlight the correct URL version in your answer. I was a bit confused at the beginning because three out of four URLs in this post use the wrong format. – florestan Jan 21 '21 at 11:35
  • What are valid schemes for `://`? As @kerolos asked, I also need to install from a local package directory and can't figure out the correct scheme. Perhaps link to pip source you're referencing in your answer? – jacob Mar 03 '23 at 16:32
  • 1
    Never mind, I came up with a solution to local package dependencies [here](https://stackoverflow.com/a/75630599/9403538). – jacob Mar 03 '23 at 18:24