11

I have a local GitLab installation that comes with a local PyPI server to store company internal Python packages.

How can I configure my PyPI to search packages in both index servers?

I read about .pypirc / pip/pip.ini and found various settings but no solution so far.

  1. Most solutions permanently switch all searches to the other index server. But I want to be able to install and update packages from pypi.org as normal while some packages come from the local index.
  2. setting multiple index servers with credentials seams to be limited to distutils (used e.g. by twine) only, but is not used by pip
  3. There is confusion if to configure index servers in [global] or [install]. I assume the latter one is a rule subset for pip install. (The documentation is here unclear.)
  4. While twine can reference a repository entry in the config file like -r gitlab refers to a [gitlab] section, such a named reference can't be used by pip...

So what I want to achieve:

  • pip should be able to install and update regular packages from pypi.org like colorama
  • pip should be able to install and update packages from gitlab.company.com
    • authentication with username (__token__) and password (7a3b62342c784d87) must work

Experiment so far:

[global]


[install]
find-links =
    https://pypi.org
    https://gitlab.company.de/api/v4/projects/2142423/packages/pypi
trusted-host =
    https://pypi.org
    https://gitlab.company.de/api/v4/projects/2142423/packages/pypi

[distutils]
index-servers =
    gitlab

[gitlab]
repository = https://gitlab.company.de/api/v4/projects/2142423/packages/pypi
username = __token__
password = geheim
Paebbels
  • 15,573
  • 13
  • 70
  • 139
  • 2
    When you say "search" do you literally mean the result from `pip search`? Or just want gets installed from `pip install`? If it's the latter, https://stackoverflow.com/questions/30889494/can-pip-conf-specify-two-index-url-at-the-same-time should answer this for you. – Dustin Ingram Sep 08 '20 at 22:46
  • "search" refers to searching in general, not the `pip search` command. – Paebbels Sep 09 '20 at 04:11
  • @DustinIngram I assume you refer to `--extra-index-url`. But how does it handle the credentials? I see only a URL. I don't want to enter or copy/paste on every `pip ***`command a long token by hand. – Paebbels Sep 09 '20 at 04:14
  • I think you are looking for something like this --> https://devpi.net/docs/devpi/devpi/stable/%2Bd/index.html – gold_cy Sep 29 '20 at 01:18

3 Answers3

10

Goal

  1. pip install should install/update packages from GitLab as well as PyPi repo. If same package is present in both, PyPi is preferred.
  2. pip install should support authentication. Preferred, if somehow we can make it read from a config file so that we don't need to specify it repeatatively.

Theory

  1. pip install supports --extra-index-url to specify additional PyPi indexes. The same can also be provided via pip.conf file.
  2. pip uses requests which supports ~/.netrc as config file (docs).

Steps

  1. Create a pip.conf (pip.ini if on Windows) in any of the locations suggested by pip config -v list.
  2. Add your GitLab PyPi index URL to pip.conf.
[install]
extra-index-url = https://gitlab.com/api/v4/projects/12345678/packages/pypi/simple
  1. Create or update your ~/.netrc file and add your auth details for GitLab.
machine gitlab.com
    login <token-name>
    password <token-pass>
  1. We can now install packages as simply as pip install <package-name>. pip will now look at both indexes to find your packages, with preference provided to the one pointed by index-url.

Additional info

  1. The same could have been possible for pip search too, had there been support for multiple indexes. Till then, one needs to manually specify which PyPi index URL should be used. GitLab does not seem to support pip search since it throws 415 Client Error: Unsupported Media Type when specified as the PyPi index.
  2. As for your doubts, each section in pip.conf points to that particular command, [install] provides configuration for pip install, [search] for pip search and so on. [global] probably refers to parameters that can be specified for all the commands be it pip install or pip search.
  3. .pypirc file is made specially for configuring package indexes related to upload (used by twine/flint), where as pip.conf is associated with configuring pip which manages python packages on your local system.
wim
  • 338,267
  • 99
  • 616
  • 750
Amit Singh
  • 2,875
  • 14
  • 30
  • 2
    The problem with pip search is that it uses deprecated XML-RPC api, and that never got changed despite having [an open issue since 2011](https://github.com/pypa/pip/issues/395). And maybe the command [will be removed entirely](https://github.com/pypa/pip/issues/5216) since it seems nobody was really bothered enough to fix it. – wim Oct 04 '20 at 17:25
3

Try this (based on information from https://github.com/pypa/pip/issues/6797 and Can pip.conf specify two index-url at the same time?):

[global]
index-url = http://pypi.org/simple
trusted-host = pypi.org
               gitlab.company.de
extra-index-url= https://username:password@gitlab.company.de/api/v4/projects/2142423/packages/pypi
Seth
  • 2,214
  • 1
  • 7
  • 21
0

It may be simpler to use the PIP_EXTRA_INDEX_URL environment variable to set multiple indices. As noted here, you can also have more than one extra index by using space delimitation between the indices.

PIP_EXTRA_INDEX_URL="https://__token__:${GITLAB_TOKEN}@gitlab.company.com/api/v4/projects/1/packages/pypi https://__token__:${GITLAB_TOKEN}@gitlab.company.com/api/v4/projects/2/packages/pypi"
3ch01c
  • 2,336
  • 2
  • 17
  • 15
  • Thanks for sharing this variable name and how it can be used with multiple entries. But it's not a direct solution to the question. My intention is to setup the new package source once and then pip install can be used. So coworkers can install from our local GitLab server without typing long commands or passwords for every install and especially for every update. – Paebbels Jul 20 '23 at 20:12
  • Sorry for not clarifying this. Commands like `pip install` will look for the `PIP_EXTRA_INDEX_URL` environment variable, similar to how they look for configuration in a `pip.ini` file. In other words, if your coworkers set up this variable in their environment, they just have to do `pip install -r requirements.txt` and those indices will be included in package resolution. It may simply be a matter of preference of how you want to pass around configuration. – 3ch01c Jul 21 '23 at 16:09