When does a airflow dag definition get evaluated?

Question

Suppose I have an airflow dag file that creates a graph like so...

def get_current_info(filename)
    current_info = {}
    <fill in info in current_info relevant for today's date for given file>
    return current_info

files = [
    get_current_info("file_001"),
    get_current_info("file_002"),
    ....
]

for f in files:
    <some BashOperator bo1 using f's current info dict>
    <some BashOperator bo2 using f's current info dict>
    ....

    bo1 >> bo2
    ....

Since these values in the current_info dict that is used to define the dag changes periodically (here, daily), I would like to know by what process / schedule the dag definition gets updated. (I print the current_info values each run and values appear to be updating, but curious as to how and when exactly this happens).

When does a airflow dag definition get evaluated? referenced anywhere in the docs?

score 1 · Accepted Answer · answered Oct 29 '19 at 06:13

1

The DAGs are evaluated in every run of the scheduler.

This article describes how the scheduler works and at what stage the DAG files are picked up for evaluation.

answered Oct 29 '19 at 06:13

GTO

56
1
3

score 1 · Answer 2 · answered Jul 10 '20 at 22:22

After some discussion on the [airflow email list][1], it turns out that airflow builds the dag for each task when it is run (so each tasks includes the overhead of building the dag again (which in my case was very significant)).

See more details on this here: https://stackoverflow.com/a/59995882/8236733

When does a airflow dag definition get evaluated?

2 Answers2

Linked