2

I configured my DAG like this:

default_args = {
    'owner': 'Aviv',
    'depends_on_past': False,
    'start_date': datetime(2017, 1, 1),
    'email': ['aviv@oron.com'],
    'email_on_failure': False,
    'email_on_retry': False,
    'retries': 0,
    'retry_delay': timedelta(minutes=1)
}

dag = DAG(
    'MyDAG'
    , schedule_interval=timedelta(minutes=3)
    , default_args=default_args
    , catchup=False
)

and for some reason, when i un-pause the DAG, its being executed twice immediatly. Any idea why? And is there any rule i can apply to tell this DAG to never run more than once in the same time?

Aviv Oron
  • 505
  • 1
  • 6
  • 10

2 Answers2

2

You can specify max_active_runs like this:

dag = airflow.DAG(
    'customer_staging',
    schedule_interval="@daily",
    dagrun_timeout=timedelta(minutes=60),
    template_searchpath=tmpl_search_path,
    default_args=args,
    max_active_runs=1)

I've never seen it happening, are you sure that those runs are not backfills, see: https://stackoverflow.com/a/47953439/9132848

x97Core
  • 1,454
  • 13
  • 20
0

I think its because you have missed the scheduled time and airflow is backfilling it automatically when you ON it again. You can disable this by catchup_by_default = False in the airflow.cfg.

Duwa
  • 76
  • 4