Flask virtual environment and environment variables

Question

To get gunicorn under supervisord to use the virtual environment /home/ubuntu/venv/bin it is not necessary to seek a judicious place to put source /home/ubuntu/venv/bin/activate. It is sufficient to write:

[program:hello]
command=/home/ubuntu/venv/bin/gunicorn -b localhost:8000 hello:app
directory=/home/ubuntu/hello/
environment=PATH="/home/ubuntu/venv/bin:%(ENV_PATH)"

in /usr/supervisor/hello.conf.

The next task is to bring in a whole slew of environment variables. One way is to laboriously augment the supervisord config file as follows.

[program:hello]
...
environment=PATH="/home/ubuntu/venv/bin:%(ENV_PATH)",SECRET_KEY="%(ENV_SECRET_KEY)",DATABASE_URI="%(ENV_DATABASE_URI)",etc1,etc2,etc3

Is there a way to bring in the environment variables in one shot (after they're initialized in, say, ~/.profile?

Related: 1, 2, 3, 4, 5, 6, 7

Does `supervisord` inherit your environment when it starts, or does it create its own? Entering the venv just sets some environment variables, so you need to ensure those are properly set in `supervisord`'s environment. — chepner, Mar 31 '19 at 03:58
@SuperShoot Ah! good point. No, it doesn't. Can you suggest another lead on the road to debugging this? — Calaf, Mar 31 '19 at 03:59
@chepner What do you mean by "inherit the enviroment"? Regardless of whether the (bash) shell is or is not in the venv, it fails. `supervisord`'s `environment` is, IIUC, for environment variables, not for a Python virtual environment. — Calaf, Mar 31 '19 at 04:01
Entering a Python virtual environment does little more than add the appropriate directory to `PATH` in your environment. If `supervisord`'s `PATH` does not include `/home/ubuntu/venv/bin`, then it won't be using the correct Python interpreter. — chepner, Mar 31 '19 at 04:03
@chepner Interesting.. so that means that I can replace any attempt at `source ..activate` with a `supervisord` environment setting inside the `[program:hello]` block, no? That does not help. — Calaf, Mar 31 '19 at 04:10
What if you return `os.environ['PATH']` from the view so you can inspect it? — SuperShoot, Mar 31 '19 at 04:13
I see that this (https://stackoverflow.com/a/1883251/704972) method (which otherwise works correctly) is misleading in this case. If I return `sys.version`, I see that I am indeed in the virtual environment in this toy/hello application, but not in my actual application. So my isolation for the purpose of this question is not complete. — Calaf, Mar 31 '19 at 04:27
@SuperShoot os.environ['PATH'] shows the venv's path. That's a good starting point to see what's happening in the bigger app. — Calaf, Mar 31 '19 at 04:31
@hd1 and not use a process controller at all? If you are using and are happy with a competitor (http://supervisord.org/glossary.html#term-daemontools), please share your experience. — Calaf, Mar 31 '19 at 21:52

Calaf · Accepted Answer · 2019-04-01T00:28:10.920

Here is a recipe:

Write the environment variables in a file /home/ubuntu/prog/.env.

export FLASK_APP=/home/ubuntu/prog/hello.py
export SECRET_KEY=ABCD
export DATABASE_PASSWORD=EFGH

Use dotenv's load_dotenv to load the environment variables.

from flask import Flask
from os.path import join, dirname
from os import environ
from dotenv import load_dotenv

app = Flask(__name__)
dotenv_path = join(dirname(__file__), '.env')
load_dotenv(dotenv_path)

@app.route('/')
def hello():
    SECRET_KEY = environ.get("SECRET_KEY")
    DATABASE_PASSWORD = environ.get("DATABASE_PASSWORD")
    return SECRET_KEY + DATABASE_PASSWORD

Write a file /etc/supervisor/hello.conf.

[program:hello]
command=/home/ubuntu/venv/bin/gunicorn -b localhost:8000 hello:app
directory=/home/ubuntu/prog
stdout_logfile=/home/ubuntu/prog/hello_out.log
stderr_logfile=/home/ubuntu/prog/hello_err.log
user=ubuntu
autostart=true
autorestart=true
stopasgroup=true
killasgroup=true

[supervisord]
logfile=/home/ubuntu/prog/hello_supervisord.log
pidfile=/tmp/supervisord.pid

Load the environment and point to the the app.

source /home/ubuntu/prog/.env

The environment variables are now loaded,

$ export | grep SECRET
declare -x SECRET_KEY="ABCD"

and they will be passed to the sub-process without messing with supervisord's environment=.

Launch supervisord in the foreground to confirm all is well.

/usr/bin/supervisord -n -edebug -c /etc/supervisor/hello.conf

Confirm from another shell that all is well.

$ curl localhost:8000
ABCDEFGH

Kill supervisord. Since it's in the foreground, it's enough to CTRL-c it.

Launch supervisord as a daemon.

/usr/bin/supervisord -c /etc/supervisor/hello.conf

Keep an eye on the three log files prog/hello_out.log, prog/hello_err.log, and prog/hello_supervisord.log.

Perhaps the most important point is to avoid using supervisord's environment=. SO chatter suggests that it handles commas, quotation marks, tabs, even newlines. Empirically, this doesn't hold (at least for supervisord 3.3.5), and the documentation does not settle it one way or the other. The two lines 942-943 seem to be where the parsing happens, if someone cares to investigate the insufficiency of the docs.

score 0 · Answer 2 · answered Nov 11 '19 at 16:33

To bring in the environment variables in one shot, permanently, securely you have to add following lines to .bashrc in $HOME directory. For this open the .bashrc file in your home directory with your favorite code editor:

nano .bashrc

Add the following line somewhere in your .bashrc file:

export SECRET_KEY="YOUR SECRET KEY."

Now to use this SECRET_KEY in flask you have to import os module and make use of it like this.

from flask import Flask
import os

app = Flask(__name__)

@app.route('/')
def hello():
    SECRET_KEY = os.environ.get('SECRET_KEY')
    return SECRET_KEY

Hope this helps.

Flask virtual environment and environment variables

2 Answers2