19

I have a privately hosted package that has a name conflict with a public package found on PyPi. Unfortunately, because the public version is higher than my private package, a simple pip install <package_x> command finds the public version instead of my privately hosted package.

In effect:

PyPi (public) hosts package_x==1.5.0
PrivateRepo (private) hosts package_x==1.3.0

I would like pip install package_x to install the private version 1.3.0 without requiring me to specify the version or the index within the pip install command (purely through a configuration file.)

I'm trying to set up my pip config to look only at a specific private repo for a single package, but both the private repo and the standard https://pypi.python.org/simple/ repo for everything else. I tried setting the private repo as my index-url and PyPi as an extra-index-url, but that will still search both repositories for the most recent package version.

Is there anyway to specify, within my pip config, the specific repo to use for a given package? Ideally something like this:

[global]
force-index: https://privaterepo.net/simple
    - package_x
Martijn Pieters
  • 1,048,767
  • 296
  • 4,058
  • 3,343
Brandon Schabell
  • 1,735
  • 2
  • 10
  • 21
  • 2
    Pipenv supports this, and I suspect Poetry does too. Are you married to `pip`, or are you willing to switch to a newer tool? – ChrisGPT was on strike May 13 '20 at 19:42
  • 1
    I am fairly married to pip, unfortunately. I'm trying to set this up in an enterprise environment in which dozens of engineers are going to be using pip. The goal is to simplify their lives as much as possible. – Brandon Schabell May 13 '20 at 19:44
  • The ability to limit the searched repos for certain (private) packages will become vital in the very near future as demonstrated in [this article](https://medium.com/@alex.birsan/dependency-confusion-4a5d60fec610). – Tom Pohl Jul 22 '21 at 15:36
  • Using only configuration this can't be achieved. A writeup of the current state of package source selection can be found [here](https://github.com/ComputeCanada/software-stack/blob/main/pip-which-version.md). And this is the correlating issue on pip's own github repo: https://github.com/pypa/pip/issues/10156 – fabianegli Jul 28 '21 at 08:11

2 Answers2

4

If you're using Pip you can add --extra-index-url in the requirements.txt to specify additional repositories like this:

--extra-index-url=https://your-private-pypi-repo.com/simple
Flask==2.0.2
helloworld==0.1.0
crazy_p
  • 240
  • 3
  • 6
2

Pip

Unfortunately pip does not allow this. The closest you come to something like this is to use the --index-url option in a requirements.txt file to use a different index, but this is a global option which means it overrides any other such options already defined, and it cannot be defined per package

There are other recent tools available which allows us to have granular control of python dependencies and their sources

Poetry

The first is Poetry, which gives us this option within a pyproject.toml file.

From the docs

[[tool.poetry.source]]
name = "foo"
url = "https://foo.bar/simple/"
secondary = true

[tool.poetry.dependencies]
python = "^3.9"
requests = {version="*", source="foo"}
maya = {version="*", source="pypi"}
records = "*"

The above adds a new package repo called foo to your list of sources, but still keeps pypi as the primary source. The docs also explains how to completely ignore pypi and use your private repo as the only source for packages.

In the tool.poetry.dependencies section, we also specify three packages and where they should be installed from:

  • requests installs from foo
  • maya installs from pypi
  • records installs by default from pypi

Install the dependencies with:

poetry install --no-root

You can also use the --source option to the poetry add requests to specify which source to use

Pipenv

You can use Pipenv to accomplish this.

From the docs, add the following to your Pipfile:

[[source]]
url = "https://pypi.org/simple"
verify_ssl = true
name = "pypi"

[[source]]
url = "http://pypi.home.kennethreitz.org/simple"
verify_ssl = false
name = "home"

[dev-packages]

[packages]
requests = {version="*", index="home"}
maya = {version="*", index="pypi"}
records = "*"

The above, allows you to install maya from pypi, whereas requests will be installed from a custom/private package index.

smac89
  • 39,374
  • 15
  • 132
  • 179