Proper way to parse requirements file after pip upgrade to pip 10.x.x?

Question

So today I did found out that with the release of pip 10.x.x the req package changed its directory and can now be found under pip._internal.req.

Since it is common practice to use the parse_requirements function in your setup.py to install all the dependencies out of a requirements file I now wonder if this practice should change since it is now lying under _internal?

Or what is actually best practice without using parse_requirements?

Answer 1

First, I believe parsing requirements.txt to fill the list of dependencies in package metadata is not a good idea. The requirements.txt file and the list of "install dependencies" are two different concepts, they are not interchangeable. It should be the other way around, the list of dependencies in package metadata should be considered as some kind of source of truth, and files such as requirements.txt should be generated from there. For example with a tool such as pip-compile. See the notes at the bottom of this answer.

But everyone has different needs, that lead to different workflows. So with that said... There are 3 possibilities to handle this, depending on where you want your project's package metadata to be written: pyproject.toml, setup.cfg, or setup.py.

---

Words of caution!

If you insist on having the list of dependencies in package metadata be read from a requirements.txt file then make sure that this requirements.txt file is included in the "source distribution" (sdist) otherwise installation will fail, for obvious reasons.

These techniques will work only for simple requirements.txt files. See Requirements parsing in the documentation page for pkg_resources to get details about what is handled. In short, each line should be a valid PEP 508 requirement. Notations that are really specific to pip are not supported and it will cause a failure.

---

pyproject.toml

[project]
# ...
dynamic = ["dependencies"]

[tool.setuptools.dynamic]
# ...
dependencies = requirements.txt

---

setup.cfg

Since setuptools version 62.6 it is possible to write something like this in setup.cfg:

[options]
install_requires = file: requirements.txt

---

setup.py

It is possible to parse a relatively simple requirements.txt file from a setuptools setup.py script without pip. The setuptools project already contains necessary tools in its top level package pkg_resources.

It could more or less look like this:

#!/usr/bin/env python

import pathlib

import pkg_resources
import setuptools

with pathlib.Path('requirements.txt').open() as requirements_txt:
    install_requires = [
        str(requirement)
        for requirement
        in pkg_resources.parse_requirements(requirements_txt)
    ]

setuptools.setup(
    install_requires=install_requires,
)

---

Notes:

• https://github.com/pypa/setuptools/issues/1951#issuecomment-1431345869
• https://caremad.io/posts/2013/07/setup-vs-requirement/
• https://setuptools.pypa.io/en/latest/history.html#v62-6-0
• See also this other answer: https://stackoverflow.com/a/59971469

Answer 2

The solution of Scrotch only works until pip 19.0.3, in the pip >= 20 versions the PipSession module was refactored. Here is a solution for the imports that works for all pip versions:

try:
    # pip >=20
    from pip._internal.network.session import PipSession
    from pip._internal.req import parse_requirements
except ImportError:
    try:
        # 10.0.0 <= pip <= 19.3.1
        from pip._internal.download import PipSession
        from pip._internal.req import parse_requirements
    except ImportError:
        # pip <= 9.0.3
        from pip.download import PipSession
        from pip.req import parse_requirements

Answer 3

EDIT: modified to support pip>= 19.0.3

I don't agree with the accepted answer. The setup.py file can get ugly real fast if you have a large project with a lot of dependencies. It is always good practice to keep your requirements in a separate .txt file. I would do something like this -

try:
    # pip >=20
    from pip._internal.network.session import PipSession
    from pip._internal.req import parse_requirements
except ImportError:
    try:
        # 10.0.0 <= pip <= 19.3.1
        from pip._internal.download import PipSession
        from pip._internal.req import parse_requirements
    except ImportError:
        # pip <= 9.0.3
        from pip.download import PipSession
        from pip.req import parse_requirements

requirements = parse_requirements(os.path.join(os.path.dirname(__file__), 'requirements.txt'), session=PipSession())

if __name__ == '__main__':
    setup(
        ...
        install_requires=[str(requirement.requirement) for requirement in requirements],
        ...
    )

Throw in all your requirements in requirements.txt under project root directory.

Answer 4

What I figured out the right way to do is adding the dependencies in the setup.py like:

REQUIRED_PACKAGES = [
    'boto3==1.7.33'
]

if __name__ == '__main__':
    setup(
        ...
        install_requires=REQUIRED_PACKAGES,
        ...
    )

and just have a . in your requirements.txt. It will then automatically install all packages from the setup.py if you install from the file.

Answer 5

@sinoroc is correct that you generally do not want to populate your setup() function using your requirements.txt. They are for different things: requirements.txt produces a maximal, reproducible environment with hard versions to aid in deployments, help other developers etc.. Setup dependencies are a minimal, permissive list to allow users to install things.

However, sometimes you want to install an application using pip, or collaborators insist on using the requirements file, so I wrote a package which helps you do this: https://pypi.org/project/extreqs/

Answer 6

with open("requirements.txt") as f:
    dependencies = [line for line in f if "==" in line]

setup(
    install_requires=dependencies
)