2

DAG A has schedule '0 6 * * *'.

DAG B has schedule '*/5 * * * *'.

However, DAG B should only start running for that day once DAG A has completed for that day.

I've played around with SubDags and ExternalTaskSensor but haven't yet found a satisfactory solution and I'm sure I'm missing something good. Recommendations?

Edit: say DAG A is my ETL. DAG B has some tasks that query my database and require that data to be up-to-date. DAG B gets run throughout the day, but only once the ETL is completed.

I can see using ShortCircuitOperator, for example, and having the condition be "DAG A that ran today is completed." But how could I write this condition?

smartestblonde
  • 51
  • 1
  • 8
  • How was your experience with SubDAGs? Did anything not work as expected? – Taylor D. Edmiston Jun 29 '18 at 21:33
  • I looked into SubDAGs because I mistakenly though "ah, so DAG B can be triggered once DAG A completes, and then I can have SubDAG C of DAG B that will run every 5 minutes once it's kicked off" ... but then I found that the SubDAG has to run on the same schedule as its parent DAG, so I hadn't actually gained anything. – smartestblonde Jul 02 '18 at 01:36

2 Answers2

3

This question is not an exact duplicate but is similar to another which already has 3 good answers: Scheduling dag runs in Airflow.

I recommend reading through all of them, but to summarize the info in the answers over there, there are several viable options for the use case of a DAG dependent upon another DAG:

  • TriggerDagRunOperator
  • BranchPythonOperator
  • ShortCircuitOperator
  • SubDagOperator / SubDAGs
  • With any of these options you may want to experiment with the trigger rule
  • External triggers (possibly less relevant for your use case)

If you can add more detail around the use case you're trying to accomplish, I could give more specific guidance as well.

Taylor D. Edmiston
  • 12,088
  • 6
  • 56
  • 76
  • 2
    So, looking into each of these, I see that they do work if I want to trigger one dag after another dag finishes. However, what I want is DAG B to be on a schedule that runs every 5 minutes, but only to run if DAG A is completed for that day. Each of these solutions, from what I can see, trigger just one run of DAG B -- am I missing something? – smartestblonde Jul 02 '18 at 01:33
0

Use TriggerDagRunOperator for calling a DAG to run after another. Refer to this question. Afraid I cannot provide a satisfactory example as I have not used it yet.

Zack
  • 2,296
  • 20
  • 28