111

I'm using pip with virtualenv to package and install some Python libraries.

I'd imagine what I'm doing is a pretty common scenario. I'm the maintainer on several libraries for which I can specify the dependencies explicitly. A few of my libraries are dependent on third party libraries that have transitive dependencies over which I have no control.

What I'm trying to achieve is for a pip install on one of my libraries to download/install all of its upstream dependencies. What I'm struggling with in the pip documentation is if/how requirements files can do this on their own or if they're really just a supplement to using install_requires.

Would I use install_requires in all of my libraries to specify dependencies and version ranges and then only use a requirements file to resolve a conflict and/or freeze them for a production build?

Let's pretend I live in an imaginary world (I know, I know) and my upstream dependencies are straightforward and guaranteed to never conflict or break backward compatibility. Would I be compelled to use a pip requirements file at all or just let pip/setuptools/distribute install everything based on install_requires?

There are a lot of similar questions on here, but I couldn't find any that were as basic as when to use one or the other or using them both together harmoniously.

behnam
  • 1,959
  • 14
  • 21
Joe Holloway
  • 28,320
  • 15
  • 82
  • 92

4 Answers4

74

My philosophy is that install_requires should indicate a minimum of what you need. It might include version requirements if you know that some versions will not work; but it shouldn't have version requirements where you aren't sure (e.g., you aren't sure if a future release of a dependency will break your library or not).

Requirements files on the other hand should indicate what you know does work, and may include optional dependencies that you recommend. For example you might use SQLAlchemy but suggest MySQL, and so put MySQLdb in the requirements file).

So, in summary: install_requires is to keep people away from things that you know don't work, while requirements files to lead people towards things you know do work. One reason for this is that install_requires requirements are always checked, and cannot be disabled without actually changing the package metadata. So you can't easily try a new combination. Requirements files are only checked at install time.

Ian Bicking
  • 9,762
  • 6
  • 33
  • 32
  • 6
    does this means you should mirror `setup.py` `install_requires=` deps in `requirements.txt`? – proppy Sep 25 '13 at 23:24
  • 16
    Having both, requirements in setup.py and a requirements file is dangerous though, because the duplication just asks to become out of sync. – Sebastian Blask Nov 13 '13 at 16:30
  • 1
    Also, how do you actually work with it then? I'd assume, you use the requirements file once to get to a state that is definitely working. Then install with the actual package with pip. You'll never be able to use `-U` because that might override the dependencies from the requirements file? How do you upgrade? – Sebastian Blask Nov 13 '13 at 16:48
  • 1
    Does this answer apply equally to applications and packages? Imagine my-web-app (an app) depending on some-tool (a package), both of which depend on the requests package. If some-tool has a requirements.txt file that pins a particular version or version range of requests, that would seem to create a potential problem for my-web-app, which might have specified a conflicting version/version range. – Reece Mar 05 '14 at 21:53
  • 2
    There should be only way to install a package. So having both is not recommended unless you want to confuse other contributors. – Gewthen Oct 16 '15 at 21:11
  • This answer is confusing. Why would I bloat my requirements.txt file with packages that are not needed? – async Jul 11 '22 at 14:17
26

here's what I put in my setup.py:

# this grabs the requirements from requirements.txt
REQUIREMENTS = [i.strip() for i in open("requirements.txt").readlines()]

setup(
    .....
    install_requires=REQUIREMENTS
)
rbp
  • 1,850
  • 15
  • 28
  • 25
    Watch out, requirements files could contain comments and inclusions. [You should use the pip parser](http://stackoverflow.com/questions/14399534/how-can-i-reference-requirements-txt-for-the-install-requires-kwarg-in-setuptool/16624700#16624700) – Romain Hardouin Jun 10 '13 at 12:16
  • 1
    yes, I eventually changed this to strip out comments. pip parser looks better than my answer. – rbp Jun 10 '13 at 15:45
  • 7
    Why use a requirements file at all if all it contains is already in setup.py? – Sebastian Blask Nov 13 '13 at 16:27
  • 2
    @RomainHardouin, as mentioned in comments to your linked answer, pip isn't meant to be used that way. – akaihola May 22 '14 at 20:33
  • 1
    yeah this worked for me until a critical `--extra-index-url` in in the requirements was required and this blew up in my face. Thanks @RomainHardouin – Tommy Feb 08 '16 at 20:10
  • Very cool but did not solve my problem of mixed pypi and artifactory requirements. – Dave McNulla Apr 13 '20 at 23:50
  • I cannot manage to open this file when running "python -m build", requirements.txt is not found because the script is not run directly in the current folder (and __file__ is not defined either) – Eric Burel Sep 02 '22 at 20:23
20

The Python Packaging User Guide has a page about this topic, I highly recommend you read it:

Summary:

install_requires is there to list the dependencies of the package that absolutely must be installed for the package to work. It is not meant to pin the dependencies to specific versions, but ranges are accepted, for example install_requires=['django>=1.8']. install_requires is observed by pip install name-on-pypi and other tools.

requirements.txt is just a text file, that you can choose to run pip install -r requirements.txt against. It's meant to have versions of all dependencies and subdependencies pinned, like this: django==1.8.1. You can create one using pip freeze > requirements.txt. (Some services, like Heroku, automatically run pip install -r requirements.txt for you.) pip install name-on-pypi does not look at requirements.txt, only at install_requires.

Flimm
  • 136,138
  • 45
  • 251
  • 267
7

I only ever use a setup.py and install_requires because there is only one place to look at. It is just as powerful as having a requirements file and there is no duplication to maintain.

Gewthen
  • 408
  • 4
  • 12
Sebastian Blask
  • 2,870
  • 1
  • 16
  • 29
  • The question is when to use one or the other, I didn't spell it out, but my answer is saying that I always use one and never the other. How is that not answering the question? – Sebastian Blask Jan 12 '15 at 10:21