9

I found out there are lot of ways to store it as variables, hooks and other ways using encryption. I would like to know what's the best way to do it.

Raj
  • 585
  • 4
  • 16
  • 28

1 Answers1

22

Currently there 2 ways of storing secrests:

1) Airflow Variables: Value of a variable will be hidden if the key contains any words in (‘password’, ‘secret’, ‘passwd’, ‘authorization’, ‘api_key’, ‘apikey’, ‘access_token’) by default, but can be configured to show in clear-text as shown in the image below.

enter image description here

However, there is a known-bug where anyone with an access to UI can export all the variables which will expose the secrets.

2) Airflow Connections:

You can use the Passwords field in Airflow connections which will encrypt that field if you had installed the crypto package (pip install apache-airflow[crypto]). The password field would just appear as blank in the UI as shown in the screenshot.

More on Securing connections: https://airflow.apache.org/howto/secure-connections.html

enter image description here

I recommend the 2nd approach as even if someone gets access to the UI, he/she won't be able to get your secrets. Keep in mind though that you need to install the crypto package for this.

You can then access the secrets as below:

from airflow.hooks.base_hook import BaseHook

connection = BaseHook.get_connection(CONN_ID)
slack_token = connection.password

You can set the CONN_ID as the name of your connection.

kaxil
  • 17,706
  • 2
  • 59
  • 78
  • Thanks for your answer. I have to save and import multiple username and password for server, database and API. May I know the 2nd approach works? Could you please gimme a short example code of saving and importing it in the python code? – Raj Nov 05 '18 at 11:14
  • Can you link to the known security bug? I tried finding it and couldn't. – raphael Aug 28 '19 at 21:11
  • @kaxil thanks! I see you're the one with intimate knowledge of this bug :) – raphael Aug 29 '19 at 14:34
  • 5
    does anyone know if the security bug, where the passwords could be exported as variable is still a thing? We tried it out but the exported variables are encrypted ({ "password": "********" }). Could someone provide the link to the bug? – Ending Jan 30 '20 at 15:55
  • `airflow.hooks.base_hook.BaseHook` is deprecated in airflow>=2.5.3. use `from airflow.hooks.base import BaseHook` – Dave Apr 25 '23 at 13:21