323

Background

I was about to try Python package downloaded from GitHub, and realized that it did not have a setup.py, so I could not install it with

pip install -e <folder>

Instead, the package had a pyproject.toml file which seems to have very similar entries as the setup.py usually has.

What I found

Googling lead me into PEP-518 and it gives some critique to setup.py in Rationale section. However, it does not clearly tell that usage of setup.py should be avoided, or that pyproject.toml would as such completely replace setup.py.

Questions

Is the pyproject.toml something that is used to replace setup.py? Or should a package come with both, a pyproject.toml and a setup.py?
How would one install a project with pyproject.toml in an editable state?

EvgenKo423
  • 2,256
  • 2
  • 16
  • 23
Niko Föhr
  • 28,336
  • 10
  • 93
  • 96
  • 5
    See [PEP-518](https://www.python.org/dev/peps/pep-0518/). – Klaus D. Jul 19 '20 at 18:02
  • 18
    Thanks, @KlausD, that was in the top Google results but the PEP-518 did not take stance on should developers avoid using `setup.py`, and how to install the packages in editable state, if `setup.py` is not used, etc. – Niko Föhr Jul 19 '20 at 18:31

5 Answers5

142

Yes, pyproject.toml is the specified file format of PEP 518 which contains the build system requirements of Python projects.

This solves the build-tool dependency chicken and egg problem, i.e. pip can read pyproject.toml and what version of setuptools or wheel one may need.

If you need a setup.py for an editable install, you could use a shim in setup.py:

#!/usr/bin/env python

import setuptools

if __name__ == "__main__":
    setuptools.setup()
Ben Hampson
  • 23
  • 5
pce
  • 5,571
  • 2
  • 20
  • 25
  • 46
    Thanks for the shim! So it seems that the `pyproject.toml` is *almost* a replacement for `setup.py` and developers are expected to use *both* so that the project can be installed in editable state? Weirdly, I have been using `setup.py` for few years in various small projects and never had a need to remove a "chicken and egg problem". – Niko Föhr Jul 19 '20 at 18:38
  • This is also useful if you aren't use setuptools, for example [flit](https://flit.readthedocs.io/en/latest/rationale.html) or [poetry](https://python-poetry.org/) – pce Jul 19 '20 at 19:56
  • 3
    I'm confused... what does pip have to do with building? Isn't pip just for installing dependencies? – Shannon Jul 02 '21 at 02:15
  • 2
    @Shannon To build a Project you may need to install at least some abstract dependencies to execute the build system. pip can act as a installer backend, while `pip wheel target-dir` pip is acting as a build frontend and with `pip install lxml==2.4.0` as a integration fronted, see [PEP517](https://www.python.org/dev/peps/pep-0517/) – pce Jul 04 '21 at 04:56
  • 3
    A common chicken and egg problem is building an extension with pybind11. `setup.py` needs to add `pybind11.get_include()` to the compiler flags, but can't do that unless `pybind11` is installed. And if this is specified as a dependency in `setup.py` ... – ChrisD Jul 20 '21 at 21:14
111

pyproject.toml is the new unified Python project settings file that replaces setup.py. Editable installs still need a setup.py: import setuptools; setuptools.setup()

To use pyproject.toml, run python -m pip install .

Then, if the project is using poetry instead of pip, you can install dependencies (into %USERPROFILE%\AppData\Local\pypoetry\Cache\virtualenvs) like this:

poetry install

And then run dependencies like pytest:

poetry run pytest tests/

And pre-commit (uses .pre-commit-config.yaml):

poetry run pre-commit install
poetry run pre-commit run --all-files
Cees Timmerman
  • 17,623
  • 11
  • 91
  • 124
  • 4
    In what sense is `setup.py` non-editable? – Dave May 23 '21 at 15:57
  • 27
    @Dave An *editable install* is an installation where the installed files are symlinks pointing back to the source directory. This is particularly useful for developers, so they don't need to reinstall the package every time they change a line of code. – gerrit Jun 25 '21 at 07:14
  • what if I need to run `arch -arm64 pip3 install psycopg2 --no-binary :all:` AND update pyproject.toml? – Ren Apr 10 '22 at 18:44
  • @Ren https://stackoverflow.com/a/27043037/819417 with `poetry update` instead of `pip install` – Cees Timmerman Apr 15 '22 at 04:06
  • @CeesTimmerman yeah but that doesn't add the `arch -arm64` command nor the `--no-binary :all:` options. – Ren Apr 16 '22 at 05:10
  • `PIP_NO_BINARY="psycopg2"` but that's not needed according to https://github.com/python-poetry/poetry/issues/365#issuecomment-537798177 and deprecated in 1.1.0 says the thread with a workaround of editing `poetry.lock`: https://github.com/python-poetry/poetry/issues/1316#issuecomment-996865695 https://github.com/psycopg/psycopg2/issues/1286 mentions a precompiled version but at this point i'd drop stagnant Poetry or use `asyncpg`. – Cees Timmerman Apr 16 '22 at 12:33
  • Followup question, sorry if it is ignorant: I am working on adding Poetry to a large existing project and was very confused about Poetry and editable installs for a while. However, I was reminded by a coworker that development ought to be done in a virtual environment. If we assume that is true, why would a Poetry user ever need an editable install? When I make changes to package code between calls to `poetry run python3 xxx`, the changes are observed in the output. That seems sufficiently editable to me? Am I misunderstanding something? – retsigam Jan 20 '23 at 17:17
  • 1
    @retsigam That should be a separate question, but it seems that [`pip install -e`](https://www.reddit.com/r/Python/comments/t3p3ub/poetry_has_officially_fixed_permanently_included/) does the same as `poetry install` now, except for possible package list updates i don't care about as the default tool should suffice. – Cees Timmerman Jan 21 '23 at 15:58
  • 3
    Apparently, setup.py isn't needed anymore for editable installs. https://setuptools.pypa.io/en/latest/userguide/development_mode.html – Eric Duminil Mar 07 '23 at 20:44
104

What is it for?

Currently there are multiple packaging tools being popular in Python community and while setuptools still seems to be prevalent it's not a de facto standard anymore. This situation creates a number of hassles for both end users and developers:

  1. For setuptools-based packages installation from source / build of a distribution can fail if one doesn't have setuptools installed;
  2. pip doesn't support the installation of packages based on other packaging tools from source, so these tools had to generate a setup.py file to produce a compatible package. To build a distribution package one has to install the packaging tool first and then use tool-specific commands;
  3. If package author decides to change the packaging tool, workflows must be changed as well to use different tool-specific commands.

pyproject.toml is a new configuration file introduced by PEP 517 and PEP 518 to solve these problems:

... think of the (rough) steps required to produce a built artifact for a project:

  1. The source checkout of the project.
  2. Installation of the build system.
  3. Execute the build system.

This PEP [518] covers step #2. PEP 517 covers step #3 ...

Any tool can also extend this file with its own section (table) to accept tool-specific options, but it's up to them and not required.

PEP 621 suggests using pyproject.toml to specify package core metadata in static, tool-agnostic way. Which backends currently support this is shown in the following table:

enscons flit_core hatchling pdm-pep517 poetry-core setuptools
Supports
0.26.0+		 Supports
3.2+		 Supports
0.3+		 Supports
0.3.0+		 Doesn't support
Issue #3332		 Supports
61.0.0+

Does it replace setup.py?

For setuptools-based packages pyproject.toml is not strictly meant to replace setup.py, but rather to ensure its correct execution if it's still needed. For other packaging tools – yes, it is:

Where the build-backend key exists, this takes precedence and the source tree follows the format and conventions of the specified backend (as such no setup.py is needed unless the backend requires it). Projects may still wish to include a setup.py for compatibility with tools that do not use this spec.

How to install a package in editable mode?

Originally "editable install" was a setuptools-specific feature and as such it was not supported by PEP 517. Later on PEP 660 extended this concept to packages using pyproject.toml.

There are two possible conditions for installing a package in editable mode using pip:

  • Modern:
    Both the frontend (pip) and a backend must support PEP 660.
    pip supports it since version 21.3;
  • Legacy:
    Packaging tool must provide a setup.py file which supports the develop command.
    Since version 21.1 pip can also install packages using only setup.cfg file in editable mode.

The following table describes the support of editable installs by various backends:

enscons flit_core hatchling pdm-pep517 poetry-core setuptools
Supports
0.28.0+		 Supports
3.4+		 Supports
0.3+		 Supports
0.8.0+		 Supports
1.0.8+		 Supports
64.0.0+
EvgenKo423
  • 2,256
  • 2
  • 16
  • 23
  • 1
    Please explain what a `backend` is. Is `pip` a backend or a frontend? How does it relate? Where do those backends come from? Which backend is used by which frontend? – Pynchia Jun 29 '23 at 06:05
  • 3
    @Pynchia These terms are [defined by PEP 517](https://peps.python.org/pep-0517/#terminology-and-goals). In short, a _frontend_ is a tool that can initiate the build or installation process and a _backend_ is a packaging library that does the actual building. `pip` is Python's default frontend and `setuptools` is a default backend. Most other packaging tools split themselves into a frontend and a backend. Each backend and frontend may have its own features, but `pyproject.toml` support makes them compatible with each other. – EvgenKo423 Jun 30 '23 at 08:12
17

Answering this part only, as the rest has nicely been explained by others:

How would one install a project with pyproject.toml in an editable state?

Solution

Since the release of poetry-core v1.0.8 in Feb 2022 you can do this:

a) you need this entry in your pyproject.toml:

[build-system]
requires = ["poetry-core>=1.0.8"]
build-backend = "poetry.core.masonry.api"

b) run:

pip install -e .

Sources

Greg Dubicki
  • 5,983
  • 3
  • 55
  • 68
  • 4
    Update 2022: PEP 660 is implemented in `pip` (from 21.3.0) and some backends, e.g. `flit`. However no progress in `setuptools`. – kap Jan 08 '22 at 13:50
  • Note that the `poetry install` command already makes an editable install by default, which is also explained in the issue you've linked. – EvgenKo423 May 08 '22 at 13:32
  • 1
    The PR in poetry seems to be merged already. Does this means that editable versions with poetry do not require this workaround anymore? – Heberto Mayorquin Feb 19 '23 at 11:28
3

pyproject.toml can declare the files in your python package and all the metadata for it that will show in PyPi.

A tool like flit can process the pyproject.toml file into a package that can be uploaded to PyPi or installed with pip.

Other tools use pyproject.toml for other purposes. For example, pytest stores information about where to find and how to run tests, and instructions to pytest about modifying pythonpath (sys.path) before running the tests. Many IDEs can use this to help developers conveniently run tests.

user2352073
  • 31
  • 2