Django doesn't see environment variables when deployed to Elastic Beanstalk

Question

I'm trying to set up a Django/DRF application on Elastic Beanstalk, and for whatever reason, Django just can't see the desired environment variables. When I log in, I can see them just fine, by using

$ eb ssh
$ cat /opt/python/current/env

I can also see them, except relatively sensitive ones involving RDS, simply using $eb printenv.

All that appears to be set up and working properly. However, Django likes to read the environment immediately on starting up, and it appears that the environment variables just aren't set yet. I've experimented with simply inserting print(os.environ) in settings.py, and when I do that, I discover a whole bunch of environment variables which I don't need (i.e. 'SUPERVISOR_GROUP_NAME': 'httpd'), and none of the ones I've set myself, like DJ_SECRET_KEY.

I've since changed the code to report the absence of specific environment variables when it loads the settings, and from a recent run, it generated this:

[Wed Nov 23 15:56:38.164153 2016] [:error] [pid 15708] DJ_SECRET_KEY not in environment; falling back to hardcoded value.
[Wed Nov 23 15:56:38.189717 2016] [:error] [pid 15708] RDS_DB_NAME not in environment; falling back to sqlite
[Wed Nov 23 15:56:38.189751 2016] [:error] [pid 15708] AWS_STORAGE_BUCKET_NAME not in environment; falling back to local static storage.

Again, those variables are set in the settings, and they show up with any other reporting tool EB gives me. They just aren't set in time for Django to read them when it launches and reads settings.py.

This looks pretty close to this issue, but it's not really the same: I know how to see / load the environment variables into the shell when ssh-ing into the eb instance; they're just not showing up when I need them to for the actual project.

This is almost exactly the issue I'm having, but the accepted-correct answer makes no sense to me, and the top-voted answer doesn't apply; those files are already in git.

How should I configure things so that Django can see the environment variables?

Not sure about Elastic Beanstalk, but I faced similar issue while setting up my Django Projects with Apache/Nginx. And I had to set environment variables in their configuration files. May be same should be applicable here. — Moinuddin Quadri, Nov 23 '16 at 16:50
@Gustaf I'm trying to use environment variables to configure Django. For example, given an associated RDS instance, EB auto-sets several variables such as `RDS_DB_NAME`, with which one can connect to that database. The problem is that those variables are not set at the time that Django reads them. A solution might be a method by which we can delay the WSGI initialization until the environment variables are all set. They all get set eventually, but apparently not soon enough. — coriolinus, Nov 26 '16 at 19:16
I couldn't format a proper comment here so I put it as an answer instead. I hope it helps even if it is not the exact answer to your question. — Gustaf, Nov 27 '16 at 02:32

score 4 · Accepted Answer · answered Nov 28 '16 at 14:38

Given that the EB stores all these environment variables in a canonical location as a bash script, I ended up simply having bash execute the script, and updating the environment from the parsed results.

I created the file get_eb_env.py in parallel to my settings.py. Its main contents:

import os
import subprocess

ENV_PATH = '/opt/python/current/env'

def patch_environment(path=ENV_PATH):
    "Patch the current environment, os.environ, with the contents of the specified environment file."
    # mostly pulled from a very useful snippet: http://stackoverflow.com/a/3505826/504550
    command = ['bash', '-c', 'source {path} && env'.format(path=path)]

    proc = subprocess.Popen(command, stdout=subprocess.PIPE, universal_newlines=True)
    proc_stdout, _ = proc.communicate(timeout=5)

    # proc_stdout is just a big string, not a file-like object
    # we can't iterate directly over its lines.
    for line in proc_stdout.splitlines():
        (key, _, value) = line.partition("=")
        os.environ[key] = value

Then, I just import and call patch_environment() near the head of my settings.py.

that worked like a charm. Just one thing: in my EB running python 2.7.12 the `proc.communicate` does not take the `timeout` argument. Do you have any explanation? Nevertheless, you made my day. — Mattia Paterna, Aug 15 '17 at 07:58
I was running Python 3.4 for this; the `timeout` argument was introduced IIRC in Python 3.2. If you need that feature in an earlier Python, you might consider the [subprocess32](https://pypi.python.org/pypi/subprocess32/) backport. — coriolinus, Aug 16 '17 at 11:14
OMG, you're my saviour. It's 2019 and Amazon still did not fix that problem! — DataGreed, Nov 26 '19 at 00:20

score 2 · Answer 2 · answered May 04 '17 at 08:34

2

I solved the problem modifying the key "DJANGO_AWS_SECRET_ACCESS_KEY" because DJANGO generated a key with quotation marks (`) and interpreted it as a command.

answered May 04 '17 at 08:34

Sergio Martin

992
8
5

Gustaf · Answer 3 · 2016-11-28T14:29:02.440

This is not exactly what you are asking for, but I hope this solves your issue anyway.

I am using a lot of environment variables in settings. I think Elastic Beanstalk sets everything up once before it runs container commands and such and the variables are not available then and that is why your logging is showing them empty. But do you really need them the variables at that point?

You can't put whatever local development settings you need in local_settings.py and keep that out of version control.

We use them like this.

if 'DB_HOST' in os.environ:
    DATABASES = {
        'default': {
            'ENGINE': 'django.db.backends.mysql',
            'NAME': 'ebdb',
            'USER': 'ebroot',
            'PASSWORD': 'ebpassword',
            'HOST': os.environ['DB_HOST'],
            'PORT': '3306',
        }
    }

try:
    from local_settings import *
except ImportError:
    pass

They are also available when you run container_commands:

container_commands:
  01_do_something:
    command: "do_something_script_{$PARAM1}.sh"

We do need the environment variables. Your example demonstrates exactly why. I have something very similar to that in my `settings.py`, except it falls back to a Sqlite DB instead of MySql on localhost. Deploying this setup to EB _always_ ends up running Sqlite, not connecting to the shared databse properly. That's not the behavior we want for a production environment! — coriolinus, Nov 28 '16 at 09:16
Don't have an else then, we have a local settings file. Let me update my answer. — Gustaf, Nov 28 '16 at 14:26

Django doesn't see environment variables when deployed to Elastic Beanstalk

3 Answers3