3

I would like to have some tasks running every day and others running every week inside the same DAG, is it possible or should I create a different DAG for different schedule tasks ?

I have seen the solution of using ShortCircuitOperator or AirflowSkipException and I was wondering if it is a good practice or a means of bypassing limitations of Airflow. Finally my question would be : is using ShortCircuitOperator or AirflowSkipException acceptable or using a new DAG is preferable ?

Valentin Richer
  • 680
  • 1
  • 12
  • 24
  • 1
    Possible duplicate of [How to define a DAG that scheduler a monthly job together with a daily job?](https://stackoverflow.com/questions/57104547/how-to-define-a-dag-that-scheduler-a-monthly-job-together-with-a-daily-job) – y2k-shubham Jul 24 '19 at 18:08
  • Thank you for the suggestion. I have seen this solution and I was wondering if it is a good practice or a means of bypassing limitations of Airflow. Finally my question would be : is using `ShortCircuitOperator` or `AirflowSkipException` acceptable or using a new DAG is preferable ? – Valentin Richer Jul 25 '19 at 07:53
  • @ValentinRicher: Did you find an appropriate solution for this scenario? – hemant A Aug 04 '20 at 08:37
  • Yes, I think creating two separate DAGs is the best solution – Valentin Richer Aug 04 '20 at 11:54

1 Answers1

0

This is a duplicate of this.

In short, configure the task-specific start_date parameter, introduce dependencies, or use pools to segregate tasks by runtime/priority.

manesioz
  • 798
  • 1
  • 9
  • 21
  • 3
    Thank you but the solutions proposed do not apply : 1. Modifying the start_date for each task is not a good thing to do 2. Using pools to starve some tasks and favour others is not possible because it is not a question of how fast some tasks perform compared to others 3. And `priority weight` won't do the job as I need a precise schedule – Valentin Richer Jul 25 '19 at 07:46