0

I am using Airflow for a few weeks now and I use a lot xcoms to send informations from one task to another. I just figured out that these xcoms ( as the return values of the Python Operators tasks) are accessible directly from the web UI. However, I send data like credentials, etc that I don't want to be displayed in the web UI of Airflow.

I have tried to look into the airflow config file but the only option that can be changed is the possibility to pickle xcoms.

Does anyone have a solution to my problem ? Thank you

Neil Haack
  • 3
  • 3
  • 1
    check this https://stackoverflow.com/questions/46707132/how-to-delete-xcom-objects-once-the-dag-finishes-its-run-in-airflow/57685742#57685742 looks like a possible duplicate – Anup Aug 28 '19 at 07:42

2 Answers2

0

If the credentials are static, then add it in variables and call them via Variables by importing from airflow.models import Variable. The other option that comes to my mind is adding a PostgresOperator to delete the XCOM once the dag is completed, something like below,

delete_xcom = PostgresOperator(
      task_id='task_id',
      postgres_conn_id='airflow_db',
      sql="delete from xcom where dag_id=dag.dag_id and 
           task_id='your_task_id' and execution_date={{ ds }}",
      dag=dag)
Anup
  • 250
  • 1
  • 6
  • My DAG runs for like an hour so I'd prefer not to have it displayed at all during the run. The use of Variables is a good solution I guess ! I just made some searches about it and I did not know it was possible to encrypt them, thank you very much for your answer – Neil Haack Aug 28 '19 at 09:43
0

for data credentails, you can do it in different mode based on the security strategy.

  1. hide the xcom menu and pages, you can change the app.py in www folder. just comments the xcom related parts. it should be 1-3 line of codes.

  2. encrypt the xcom raw message, change you code when push or pull messages.

  3. delete the historical xcom messages as above metioned.

Yong Wang
  • 1,200
  • 10
  • 15