1

I have requirement to send notification email, once a Google Cloud Composer airflow DAG is completed. I have come to a solution and used sendGrid as suggested in Google Documents.

But the problem is API key is visible who has access of cloud composer as it is assigned to an Environment variable. When I used the same key in my local systems python program, it is sending mail and that is where I don't understand this conecept. Because if someone has this API key or someone has shared the same it is not project specific anyone can send mail using this key.

Could some one help me, how to secure sendGrid API key.

Ashish Dhamu
  • 71
  • 1
  • 3
  • Is it possible to put it in [`Connection`s](https://airflow.apache.org/docs/stable/howto/connection/index.html)? Those can be edited only by admins; you can put your API key in the `password` field, which is encrypted with [`fernet_key`](https://airflow.apache.org/docs/stable/howto/secure-connections.html) (and not visible on UI). – y2k-shubham Aug 28 '20 at 09:24
  • Have you looked at this Stack [thread](https://stackoverflow.com/questions/45280650/store-and-access-password-using-apache-airflow)? Does it help you in your research? – Nick_Kh Aug 28 '20 at 11:10
  • The solution you guys provided is working, but if a developer will use logging.info() and provide the API_KEY as message then also he will be able to see the key in airflow logging. Is there any other work around that could be used? – Ashish Dhamu Aug 31 '20 at 10:09

1 Answers1

0

Analyzing and gathering the evidence of the community contributors in the relevant Stack thread1 and thread2 I would propose you to re-consider the general approach and use JSON file to store the credentials or the other sensitive system authentication data by specifying Keyfile Path in the corresponded Airflow connection.

  • Save the API key in JSON file and store it in Airflow environment's Cloud Storage bucket (i.e. /home/airflow/gcs/data/keyfile.json);
  • Restrict access to the JSON file, via Cloud Storage ACL as per your security considerations;
  • Create a new connection in Airflow, and in the Conn Type field, select the Google Cloud Platform option and then in Keyfile Path, enter the local file path to the JSON keyfile's location;

Since Airflow connection has been created you may refer to the encrypted API key within GoogleCloudBaseHook operator via predefined extra__google_cloud_platform__key_path query parameter in the particular DAG code:

from airflow.contrib.hooks.gcp_api_base_hook import GoogleCloudBaseHook

gcp_hook = GoogleCloudBaseHook(gcp_conn_id="<your-conn-id>")
key_path = gcp_hook._get_field("extra__google_cloud_platform__key_path")
Nick_Kh
  • 5,089
  • 2
  • 10
  • 16