1

We are working with airflow. We have something like 1000+ DAGS.

To manage DAG errors we use the same on_error_callback function to trigger alerts.

Each DAG is supposed to have context information, that could be expressed as constants, that I would like to share with the alerting stack.

Currently, I am only able to send the dag_id I retrieve from the context, via context['ti'].dag_id, and eventually the conf (parameters).

Is there a way to add other data (constants) to the context when declaring/creating the DAG?

Thanks.

Axel Borja
  • 3,718
  • 7
  • 36
  • 50

2 Answers2

0

You can pass params into the context.

dag = DAG(
    dag_id='example_dag',
    default_args=default_args,
    params={
        "param1": "value1",
        "param2": "value2"
    }
)

These are available in the context

  # example task that quickly outputs the context
  start = PythonOperator(
    task_id = 'start',
    python_callable = lambda **context: print(context)
  )

  # outputs the context, in the conf dictionary you will see
  # 'params': {'param1': 'value1'},

or via jinja templates {{params.param1}}

dimButTries
  • 661
  • 7
  • 15
0

define variables within a DAG and access variables by {{}} ref

Example ( with PROJECT_ID variable)

with models.DAG(
    ...
    user_defined_macros={ "PROJECT_ID" : PROJECT_ID } 
) as dag:
    projectId = "{{PROJECT_ID}}"
Jasmine
  • 71
  • 2