3

I create my Airflow DAGs as follows:

dag = DAG(...)

But in multiple tutorials and course I see that they use the with ... as clause like this:

with DAG(...) as dag:
    # Code that will use the dag variable.

I guess this way, the DAG instances will be destroyed after the scheduler executes the code block, but is there a real benefit in doing so? I can't find any documentation talking about this.

Stephen
  • 8,508
  • 12
  • 56
  • 96
Omegaspard
  • 1,828
  • 2
  • 24
  • 52
  • 2
    You can see what `__enter__` and `__exit__` do in the source code: https://github.com/apache/airflow/blob/master/airflow/models/dag.py – jonrsharpe Feb 07 '20 at 13:32
  • 1
    @jonrsharpe I see that the the dag is pushed to a dag context manager, which seems to be some kind of collection. Does that mean are supposed to use the "with ... as " clause when creating a dag ? – Omegaspard Feb 07 '20 at 13:35
  • 1
    Reason is mentioned [here](https://stackoverflow.com/a/1369553/10981911) – manesioz Feb 07 '20 at 14:04
  • 2
    I understand what the "with as" clause do but I was rather asking the impact on airflow ecosystem. – Omegaspard Feb 07 '20 at 14:06
  • 2
    One little advantage is that you don't have to type in each task dag=dag – Javier Lopez Tomas Feb 07 '20 at 14:31

1 Answers1

6

Yes.

If you understand what with...as does, then you should understand that its impact on the airflow ecosystem is really no different.

Specifically, it ensures that unmanaged resources -in this case implementations of the DAG class- are properly cleaned up, even if there are exceptions thrown (without needing to use a try/except block every time.)

Additionally it's nice to not have to add dag=dag to every single one.

Forest
  • 808
  • 6
  • 9
  • 1
    What actually needs to be cleaned up, though? `with` is used a lot to make sure that open file handles are automatically closed, for example. What would an Airflow DAG need to make sure was cleaned up? – Stephen Oct 13 '21 at 13:48