0

I need to run a DAG exactly once, but waiting 10 minutes before:

with models.DAG(
        'bq_executor',
        schedule_interval = '@once',
        start_date= datetime().now() + timedelta(minutes=10) ,
        catchup = False,
        default_args=default_dag_args) as dag:
       // DAG operator here

but I can't see the execution after 10 minutes. Something wrong with start_date?

If I use schedule_interval = '*/10 * * * *' and start_date= datetime(2019, 8, 1) (old date from now), I can see the excution every 10 minutes

C.P.O
  • 1,213
  • 2
  • 11
  • 29
  • 1
    It is [strongly recommended](https://stackoverflow.com/a/41194998/3679900) that you DON'T use `datetime.now()` in `start_date` – y2k-shubham Sep 10 '19 at 17:56

2 Answers2

2

Dont use datetime.now() as it will keep on changing whenever the DAG is loaded and now() + 10 minutes will always be a future timestamp resulting in DAG never getting scheduled.

paganinoff
  • 115
  • 3
  • 8
1

Airflow runs the DAGS you have added always AFTER the start_date. So if you have start_date today, it will start after today 23:59.

Scheduling is tricky for this, so check the documentation and examples: https://airflow.apache.org/scheduler.html

In you case, just switch the start_date to yesterday (or today -1) and it will start today with yesterday's ds (date stamp)

Jonay C. P.
  • 31
  • 1