2

I am creating a dag file, with multiple SimpleHttpOperator request. I need to skipped the next task if previous task returned a failed status. Only continue with success status.

Tried with BranchPythonOperator, which inside i will decide which task to run next. But seem it is not working.

sample of request_info will return
{
    "data":{
    "name":"Allan",
    "age":"26",
    "gender":"male",
    "country":"California"
  },
    "status":"failed"
}

request_info = SimpleHttpOperator(
    task_id='get_info',
    endpoint='get/information',
    http_conn_id='localhost',
    data=({"guest":"1"})
    headers={"Content-Type":"application/json"},
    xcom_push=True,
    dag=dag
)

update_info = SimpleHttpOperator(
    task_id='update_info',
    endpoint='update/information',
    http_conn_id='localhost',
    data=("{{ti.xcom_pull(task_ids='request_info')}}")
    headers={"Content-Type":"application/json"},
    xcom_push=True,
    dag=dag
)

skipped_task = DummyOperator(
    task_id='skipped',
    dag=dag
)
skip_task = BranchPythonOperator(
    task_id='skip_task',
    python_callable=next_task,
    dag=dag
)
def next_task(**kwangs):
    status="ti.xcom_pull(task_ids='request_info')"
    if status == "success":
        return "update_info"
    else:
        return "skipped_task"
    return "skipped_task"

request_info.set_downstream(skip_task)
#need set down stream base on ststus

I expect the flow should be, after getting the info. Identify status, if success, proceed update else proceed skipped.

Asphyxia
  • 21
  • 1
  • 3

1 Answers1

1

Generally tasks are supposed to be atomic, which means that they operate independently of one another (besides their order of execution). You can share more complex relations and dependencies by using XCom and airflow trigger rules.

manesioz
  • 798
  • 1
  • 9
  • 21
  • yes i can able to use xcom to get the value i needed (status). But how we can decide the next dag(task) base on the previous xcom(status)? – Asphyxia Jul 19 '19 at 03:50
  • [This](https://stackoverflow.com/a/43680277/10981911) answer here sufficiently answers that. Good luck – manesioz Jul 22 '19 at 13:20