9

We are planning to switch from managing airflow ourselves to Managed Apache Airflow services of AWS. Our original dags use some custom environment variables that need to be set in Managed airflow as well. So far I was not able to find a way to set custom environment variables while setting up airflow environment in MWAA. Please let me know if anyone knows how to set them.

Vibhav
  • 181
  • 5
  • 11
  • Is https://docs.aws.amazon.com/mwaa/latest/userguide/configuring-env-variables.html what you need? – Elad Kalif Jun 09 '21 at 07:18
  • 1
    Yes. But here we can only choose from the available configurations. There is a dropdown that shows up listing configuration options. We can configure only those options. But if I want to define a configuration option by myself as we can do in normal airflow configurations, Ex. AIRFLOW__CORE__MYCONFIG = 'something' I am not able to see how we can do that here. I require this for 2 purposes, one to directly use them in DAGs and the second, I need to set up ENV variables for my non-DAG scripts in plugins. – Vibhav Jun 09 '21 at 13:48

3 Answers3

6

An approach for setting environment variables is to use Airflow Variables.

In MWAA, you can store Airflow Variables in AWS Secrets Manager. This approach is documented in MWAA's official documentation. Note that this approach requires specific configuration for the MWAA environment. Once the MWAA environment is set up and the variables are stored in AWS Secrets Manager, the variables become accessible through the Airflow Variable APIs.

from airflow.models import Variable

# Normal call style
foo = Variable.get("foo")

See Step two: Create the Secrets Manager backend as an Apache Airflow configuration option
See Step four: Add the variables in Secrets Manager

Andrew Nguonly
  • 2,258
  • 1
  • 17
  • 23
2

you should be able to set core.myconfig env variable. mwaa accepts it even though its not on the list. mwaa will create AIRFLOW__CORE__MYCONFIG env variable.

Eziott
  • 21
  • 1
  • Thanks. I was trying to avoid doing that as the Environment variables necessarily have AIRFLOW_SECTION__ as a prefix to them if we do that. Our common files already have specific environment variable names without the prefix of AIRFLOW__SECTION__. But it seems like there is no direct way to set them as we like it. So finally we have decided to go ahead with this approach and change our common files. – Vibhav Jul 09 '21 at 17:15
  • perhaps this one is better: https://docs.aws.amazon.com/mwaa/latest/userguide/samples-env-variables.html – Eziott Jul 14 '21 at 18:23
  • 1
    Already tried this. The ENV variables created here work for the dag files but not for non-dag files. – Vibhav Jul 16 '21 at 11:28
0

You could also create a custom plugin that generates runtime environment variables.

Link Ref: https://docs.aws.amazon.com/mwaa/latest/userguide/samples-env-variables.html

Priyadarshan Mohanty
  • 137
  • 2
  • 6