2

Having list of tasks which calls different dags in master dag.I'm using the TriggerDagrunoperator to accomplish this. But facing few issues.

  • TriggerDagrunoperator doesn't wait for completion of external dag, it triggers next task. I want that to wait until completion and next task should trigger based on the status. Came across ExternalTaskSensor. It is making the process complicated. Is there any other solution to fix this?

  • If I trigger the master dag again, I want the task to restart from where it is failed. Right now, it's not restarting, but for time based schedule,it will.

Raja
  • 507
  • 1
  • 6
  • 24

2 Answers2

3

.. I want that to wait until completion .. Came across ExternalTaskSensor. It is making the process complicated ..

I'm unaware of any other way to achieve this. I myself did this the same way.

If I trigger the master dag again, I want the task to restart from where it is failed...

This requirement of your goes against the principle of idempotency that Airflow demands. I'd suggest you try to re-work you jobs in incorporate idempotency (for instance in case of retries, you have to have idempotency). Meanwhile you can take inspiration from some people and try to achieve something similar (but it will be pretty complicated)

y2k-shubham
  • 10,183
  • 11
  • 55
  • 131
  • I have implemented ExternalTaskSensor.Thanks – Raja Jul 15 '19 at 09:05
  • Master dag triggers list of dags. Dependency is set in between child dags. For instance, if any dag failed in between, I want to start from failed one. Don't want to start from initial child dag. Doing so, we can save ETL processing time. Any other ways to accomplish this? Thanks – Raja Jul 15 '19 at 09:13
  • This is what i have implemented. In Master Dag, one task(triggerdagrunoperator) will trigger the child dag and another task(externaltasksensor) will wait for child dag completion. Any ways to poke the db after x minutes. Why because, if child dag completes in 15 mins. I dont want to poke starting from 0th minutes. I guess it will occupy the resources while poking. Instead it should wait for some minimum threshold time,(say 10 mins) then poking should happen. – Raja Jul 15 '19 at 09:20
0

With Airflow 2.0.1, the triggering dag can be made to wait for completion of target dag with parameter wait_for_completion

ref: here

khari-sing
  • 650
  • 6
  • 16