179

I have two branches, Development and Production. Each has dependencies, some of which are different. Development points to dependencies that are themselves in development. Likewise for Production. I need to deploy to Heroku which expects each branch's dependencies in a single file called 'requirements.txt'.

What is the best way to organize?

What I've thought of:

  • Maintain separate requirements files, one in each branch (must survive frequent merges!)
  • Tell Heroku which requirements file I want to use (environment variable?)
  • Write deploy scripts (create temp branch, modify requirements file, commit, deploy, delete temp branch)
Charles R
  • 17,989
  • 6
  • 20
  • 18

3 Answers3

347

You can cascade your requirements files and use the "-r" flag to tell pip to include the contents of one file inside another. You can break out your requirements into a modular folder hierarchy like this:

`-- django_project_root
|-- requirements
|   |-- common.txt
|   |-- dev.txt
|   `-- prod.txt
`-- requirements.txt

The files' contents would look like this:

common.txt:

# Contains requirements common to all environments
req1==1.0
req2==1.0
req3==1.0
...

dev.txt:

# Specifies only dev-specific requirements
# But imports the common ones too
-r common.txt
dev_req==1.0
...

prod.txt:

# Same for prod...
-r common.txt
prod_req==1.0
...

Outside of Heroku, you can now setup environments like this:

pip install -r requirements/dev.txt

or

pip install -r requirements/prod.txt

Since Heroku looks specifically for "requirements.txt" at the project root, it should just mirror prod, like this:

requirements.txt:

# Mirrors prod
-r requirements/prod.txt
Christian Abbott
  • 6,497
  • 2
  • 21
  • 26
  • 2
    You ignored the problem of how to use separate requirements files for different environments on heroku. – Ed J Feb 26 '16 at 03:23
  • 51
    I believe my answer addressed that. – Christian Abbott Mar 27 '16 at 09:00
  • 1
    I was looking for a way to have different requirements on Heroku for staging (where I want additional debug packages) and production environment (where I don't need these debug packages). Unfortunately, as @EdJ said, this answer does not address this. – Antoine Pinsard Dec 17 '16 at 20:55
  • 2
    I could be misunderstanding your question or perhaps your question is different than the original poster. But for clarification, the staging branch's requirements.txt can contain "-r requirements/staging.txt" (or similar), while the prod branch's one can contain "-r requirements/prod.txt" (see end of my answer). Sync the appropriate branch to its corresponding Heroku instance. – Christian Abbott Dec 18 '16 at 22:31
  • 5
    When I install a new package using `pip install foobar`, how do I tell pip to put `foobar` as a dependency only in the `requirements/production.txt`? Or do I have to manually put `foobar` in the `requirements/production.txt` file? – Manan Mehta May 30 '18 at 23:17
  • @Manan Mehta You have to say it explicitly to which file you want to store your dependency. – Soham Navadiya Aug 20 '18 at 19:49
  • @SohamNavadiya What's the flag to specify that? I couldn't find how to put a dependancy in the requirements on `pip install` – Manan Mehta Aug 27 '18 at 21:23
  • @MananMehta `pip freeze > requirements/production.txt` – Soham Navadiya Aug 28 '18 at 06:34
  • 5
    @SohamNavadiya That is not what I asked. Let's say I have a `base.txt` with 3 packages in it, and `dev.txt` with 1 package in it (and `-r base.txt`). ALL 4 packages are installed in my virtual environment. I want to now install the 5th package and list it in base, NOT in dev, how do I do it? Sure, I can install it and `pip freeze > base.txt` but that does NOT solve the problem. It then puts the 4th dev dependency in base which I do not want. – Manan Mehta Aug 29 '18 at 00:10
  • Since this answer is now nearly 5 years old, it's worth mentioning in response to @SohamNavadiya that using pipenv over pip for package management is now a viable alternative with easier built-in support for separate dev and prod environments. For example, with pipenv, installing a new package just to prod is `pipenv install ` and for dev `pipenv install --dev` (pipenv then manages the separate environments for you without manual maintenance, plus makes use of a lock file). Pipenv docs: https://pipenv.readthedocs.io/en/latest/ – Christian Abbott Aug 29 '18 at 06:08
  • 1
    @dwanderson Apologies for not being clear. I meant I want to *list* it in base only (I don't want to include the dependency in dev.txt - BUT I understand that it is a dependency for the dev environment). So my question really is, how, during `pip install` can I specify a requirements file to list the package in, as `pip freeze` wouldn't work to segregate the dependencies between dev and base. Please re-read my above comment and see if you could understand the issue with using `pip freeze` – Manan Mehta Aug 30 '18 at 22:10
  • @dwanderson Paraphrasing, I have activated the `dev` environment on my local machine, I have two files in the project; `base.txt` and `dev.txt` and they both have some packages listed. Let's say there is a package `x` in `dev.txt` that is not in `base.txt`. I now want to install a new package `y` for *all* environments and thus I want to list it in `base.txt`, so I run `pip install y`. NOW, I cannot `pip freeze > base.txt` because my environment includes package `x` that isn't supposed to be in base, as it is a dev dep. Is the solution to maintain a separate env for each requirements file? – Manan Mehta Aug 30 '18 at 22:19
  • @dwanderson or is there an equivalent to `pip install y --target base.txt`? It isn't really that tough to understand the problem. – Manan Mehta Aug 30 '18 at 22:20
  • @ChristianAbbott Thanks, that's a great tool. Just wanted to know if its possible to use just `pip` to achieve the same thing but I guess not. – Manan Mehta Aug 30 '18 at 22:21
  • @dwanderson I was not trying to be condescending. English is not my first language, so its tough to explain a point when I have limited characters, sorry. – Manan Mehta Sep 07 '18 at 20:09
22

A viable option today which didn't exist when the original question and answer was posted is to use pipenv instead of pip to manage dependencies.

With pipenv, manually managing two separate requirement files like with pip is no longer necessary, and instead pipenv manages the development and production packages itself via interactions on the command line.

To install a package for use in both production and development:

pipenv install <package>

To install a package for the development environment only:

pipenv install <package> --dev

Via those commands, pipenv stores and manages the environment configuration in two files (Pipfile and Pipfile.lock). Heroku's current Python buildpack natively supports pipenv and will configure itself from Pipfile.lock if it exists instead of requirements.txt.

See the pipenv link for full documentation of the tool.

Christian Abbott
  • 6,497
  • 2
  • 21
  • 26
  • pipenv is a nice option but forces you to use virtualenv instead of the built in python venv. Unfortunately virtualenv breaks things like autocompletion when you are running python interactively. – Douglas Plumley Nov 02 '18 at 15:38
  • The phrase "forces you to use..." might be a bit of a strong way to put it, though. The developers would prefer pipenv work with both, however there are incompatibilities between them preventing that intention from moving forward, at least for the time being. More info here: https://github.com/pypa/pipenv/issues/15 – Christian Abbott Nov 04 '18 at 07:40
  • 28
    pipenv is a waste of time. Locking takes too long. – nurettin Jan 07 '19 at 08:59
  • 37
    pipenv is broken at almost all aspects. It promises a lot, but ships very few – ospider Feb 23 '19 at 01:24
  • 14
    @ospider Using pipenv on a daily base and am not experiencing such negative issues as you and nurettin are reporting. Working with pipenv version 2018.10.13. Broken in all aspects is thus a very empty statement. – Kwuite Mar 16 '19 at 09:46
  • 4
    @Kwuite I share the sentiment of your last sentence. There's little dialogue to engage in when a comment is critical yet vacuous. – Christian Abbott Mar 17 '19 at 10:16
  • 4
    Agree with nurettin and ospider. pipenv is awful. – Andrew Palmer Mar 31 '19 at 05:21
  • I use pipenv for macOS Catalina, and it's a great tool – nate Aug 14 '20 at 20:24
  • 1
    pipenv has its flaws, but to the best of my knowledge it is the only sane dependency + environment manager for Python to date. If conda would only properly (succinctly, sanely) update the requirements.txt file at the time of package installation, it could be better than pipenv, but it does not, AFAIK. Too bad. Python users should try Leiningen and SBT to acquire some perspective of what are good tools. Note: I did not say "perfect" tools – Trylks Oct 30 '20 at 12:56
  • 1
    Have you tried with poetry instead of pipenv? https://python-poetry.org/ – Maritza Esparza Nov 11 '20 at 20:25
  • pipenv seems fast enough these days. The (almost) infinite recursion when looking for the highest available package versions while keeping them compatible with eachother seems to be solved. So there isn't any other "vacuous criticism" left to handwave away. – nurettin Jul 12 '22 at 11:28
  • Glad you came back to make that comment @nurettin. Much agreed. – pschroeder89 Mar 15 '23 at 23:55
8

If your requirement is to be able to switch between environments on the same machine, it may be necessary to create different virtualenv folders for each environment you need to switch to. 

python3 -m venv venv_dev
source venv_dev/bin/activate
pip install -r pip/common.txt
pip install -r pip/dev.txt
exit
python3 -m venv venv_prod
source venv_prod/bin/activate
pip install -r pip/common.txt
exit
source venv_dev/bin/activate
# now we are in dev environment so your code editor and build systems will work.

# let's install a new dev package:
# pip install awesome
# pip freeze -r pip/temp.txt
# find that package, put it into pip/dev.txt
# rm pip/temp.txt

# pretty cumbersome, but it works.
nurettin
  • 11,090
  • 5
  • 65
  • 85