0

Is there a way to trigger a dag programmatically (say via API) but skip the first few tasks in the dag ?

Satfactor
  • 379
  • 1
  • 3
  • 20

1 Answers1

2

Is there a way to trigger a dag programmatically..?

I'm aware of 3 ways

1. Via an Airflow DAG itself

Use TriggerDagRunOperator

2. Via Airflow's REST API

POST /api/experimental/dags/<DAG_ID>/dag_runs

Creates a dag_run for a given dag id.

3. Airflow's CLI

Use trigger_dag command

..but skip the first few tasks in the dag?

Irrespective of whether DAG was triggered programmatically, manually via Airflow's CLI or UI, or by scheduler (normal schedule / cron time), the methods of skipping tasks are the same

1. AirflowSkipException (when you are using PythonOperator or any custom operator)

2. BranchPythonOperator or ShortCircuitOperator (these are dedicated operators for building conditional workflows)

Reference: Schedule airflow job bi-weekly

  • Please understand that you CAN'T skip a task in a DagRun just like that: to skip tasks programmatically, the logic of skipping has to be baked into the tasks / workflow itself (at the time of building / defining the dag)

  • But of course if you want to do it manually, it can be done via Airflow's UI itself (when a DAG is running) as told here

y2k-shubham
  • 10,183
  • 11
  • 55
  • 131
  • 3
    Thanks. So we found that when we trigger a task via API, we can pass a JSON which can be read in the context. then we can use this to raise the AirflowSkipException – Satfactor Sep 17 '20 at 16:33