0

I have gone through SO and web, observed this question asked previously however that question didn't answer. So asking the same question again.

Previous Question Link:

Schedule airflow job bi-weekly

How can I schedule bi-weekly jobs?

I have a requirement that I want to schedule an airflow job every alternate Friday. However, the problem is I am not able to figure out how to write a schedule for this. As I have 20 DAGS need to schedule on a bi-weekly basis, looking for a pattern.

data_addict
  • 816
  • 3
  • 15
  • 32

2 Answers2

0

The schedule_interval of a DAG supports cron expression and datetime.timedelta, it's stated in the Airflow 1.10.1 docs - I would assume this is still valid even if not mentioned in the latest docs:

schedule_interval is defined as a DAG arguments, and receives preferably a cron expression as a str, or a datetime.timedelta object.

So you should be able to set schedule_interval=timedelta(weeks=2). The following actually works, tested it with Airflow 1.10.2:

from datetime import datetime, timedelta

from airflow import DAG
from airflow.operators.bash_operator import BashOperator

default_args = {
    'task_concurrency': 3,
    'depends_on_past': True,
    'wait_for_downstream': True,
    'retries': 0,
    'start_date': datetime(2020, 9, 4),
}

SCHEDULE_INTERVAL = timedelta(weeks=2)
with DAG("bi-weekly-dag", default_args=default_args, schedule_interval=SCHEDULE_INTERVAL, catchup=True) as dag:
    t1 = BashOperator(
        task_id='bash_test',
        bash_command='echo 1',
        dag=dag
    )
Philipp Johannis
  • 2,718
  • 1
  • 15
  • 18
0

Starting Airflow 2.2.0 this is possible.

You can create a custom Timetable and customize the DAG scheduling accordingly. If you are able to define the scheduling logic by implementing next_dagrun_info and infer_manual_data_interval then Airflow would simply use this logic to schedule the task. An example can be found here.

Elad Kalif
  • 14,110
  • 2
  • 17
  • 49