poetry install different package version based on extras

Question

Using python-poetry, I would like to install different package versions based on the extras that I pass during the installation. E.g. I would like to

# when extra == 'a', install numpy == 1.20.0
$ poetry install -E a
# when extra == 'b', install numpy == 1.19.0
$ poetry install -E b

I tried it with the following toml file

[tool.poetry]
name = "demo-poetry"
version = "0.1.0"
description = ""
authors = ["tenticon"]

[tool.poetry.dependencies]
python = "^3.8"
numpy = [
    { version = "1.20.0", markers = "extra == 'a'", optional = true},
    { version = "1.19.0", markers = "extra == 'b'", optional = true}
]

[tool.poetry.extras]
a = [ "numpy" ]
b = [ "numpy" ]

[tool.poetry.dev-dependencies]
pytest = "^5.2"

[build-system]
requires = ["poetry-core>=1.0.0"]
build-backend = "poetry.core.masonry.api"

but when I do $ poetry install -E a I get

  SolverProblemError

  Because demo-poetry depends on both numpy (1.20.0) and numpy (1.19.0), version solving failed.

My poetry version is 1.1.6

I don't think this is possible. Because no one prevent you from install extra `a` and extra `b` together. And poetry checks that the dependency resolution works in every case. Is there a way to prevent installing both extras in `setuptools`? — finswimmer, May 11 '21 at 09:36
I tried adding different versions of a package using dependency groups, which are available since the pre-release 1.2.0a2 (https://github.com/python-poetry/poetry/releases/tag/1.2.0a2). However, it did not work. As indicated by @finswimmer, the groups are not mutually exclusive, and poetry checks that the dependency resolution works in every case. — vreyespue, Jan 07 '22 at 16:36

vreyespue · Accepted Answer · 2022-01-13T18:02:53.530

Poetry extras are sets of packages (e.g. a = ["numpy", "scipy"]) that can be optionally installed together with the main dependencies (poetry install -E a). When installing or specifying poetry-built packages, the extras defined in the toml file can be activated as described in PEP-508 definition of extras. Thus, the dependencies required by the a extra could be installed using pip install demo-poetry[a] (just like any other pip extras).

It is indeed possible to use environment markers as install conditions for dependencies (see list of PEP-508 environment markers). However, at the time of writing the environment marker extra is not returned by the relevant function get_marker_env(), and therefore it is ignored as install constraint. The PEP-508 documentation also notes that the extra variable is special, and that there is no current specification for it.

Neither using extras nor groups (from the pre-release 1.2.0a2) was sufficient in order to achieve the expected results. In this regard, I think the comment from @finswimmer is correct: groups are not mutually exclusive, but poetry checks that the dependency resolution works in every case.

The closest I could get to an acceptable solution was by defining conditions based on the python version, or based on the platform. Note that these conditions should always be mutually exclusive, otherwise you will get an error.

For instance, if you have the following pyproject.toml:

[tool.poetry]
name = "demo-poetry"
version = "0.1.0"
description = ""
authors = ["vreyespue"]

[tool.poetry.dependencies]
numpy = [
    { version = "1.19.0", python = "~3.7"},
    { version = "1.20.0", python = "~3.9"}
]

[build-system]
requires = ["poetry-core>=1.0.0"]
build-backend = "poetry.core.masonry.api"

You could install different versions of numpy in different environments:

$ poetry env use 3.7
  Using virtualenv: /***/demo-poetry-***-py3.7

$ poetry install
  Installing numpy (1.19.0)

$ poetry env use 3.9
  Using virtualenv: /***/demo-poetry-***-py3.9

$ poetry install
  Installing numpy (1.20.0)

score 0 · Answer 2 · answered Sep 30 '22 at 07:02

This bothered me for internal development, and while it was extremely frustrating since neither poetry nor cleo offer any documentation on their SDK, I finally managed to create a plugin for this (pypi, git).

The idea is that with this plugin, you can have non-mutual exclusive definitions in groups, and when using the --without or --only flags, the relevant groups are dropped from dependency parsing. The plugin needs to be installed prior to any such group definition.

Installation via poetry add poet-plugin (if starting from an empty, clean project, without such dependencies), or then via:

poetry shell
pip install poet-plugin

I have not battle-tested this, but it works nicely for our use case with e.g.

poetry install --only prod
poetry install --only dev
poetry install --without dev
poetry install --without prod

Feedback and updates etc are welcome, hope this momentarily resolves this issue for some of us out there.

poetry install different package version based on extras

2 Answers2