If I set env vars corresponding to airflow config settings after executing the airflow binary and at the same time DAG definitions are being loaded into memory, will this have the same effect as having set these same env vars at the OS level prior to having executed the binary?
Asked
Active
Viewed 1,518 times
1
Peter Berg
- 6,006
- 8
- 37
- 51
-
How are you setting the environment variables? – joebeeson Jun 01 '18 at 13:49
-
@joeb One way to do something like this would be to add a config.py file to you DAGs directory that, when loaded into memory, uses the `sys` package to set environment variables for the running process. – Peter Berg Jun 02 '18 at 13:55
-
Then no, that isn't the same thing as having them set at the operating system. – joebeeson Jun 02 '18 at 14:00
-
yep, that's why I asked the question :) – Peter Berg Jun 02 '18 at 14:10
1 Answers
1
I wasn't able to find any documentation on whether this would work as intended and figured that if I had to read through the source to figure this out then it's probably not a good idea to be doing it in the first place.
Instead of setting environment variables at runtime I've created two airflow.cfg files: airflow.prod.cfg and airflow.dev.cfg. I then created a shell script start.sh that cps the appropriate .cfg file to airflow.cfg prior to executing the airflow binary.
I don't love having to use the shell script to boot things up but I'd prefer that to chancing any kind of spooky action as a result of setting env vars at runtime.
Peter Berg
- 6,006
- 8
- 37
- 51