0

I am trying to see how to pass values between tasks in Airflow.

There are multiple blogs and question in SO, but most of then illustrate this using PythonOperator, where ti.xcom_push and ti.xcom_push are used within the python callable.

How can we do this inside other operators where we do not have the option of a python callable?

I tried using what is mentioned as Jinja tempaltes, but I am seeing template render errors.

Example: This works: a.

'{{ ti.xcom_pull(task_ids="get_fusion_args") }}'

But this throws a template error: b.

'{{ ti.xcom_pull(task_ids="get_fusion_args", key="fusion_args") }}'

-> I am trying to get the value pushed by a previous task using the key "fusion_args".

Why is b. throwing a template error (unable to render template)

I have tried searching for answers including blogs and posts on StackOverflow. Could not find satisfactory answers.

Reiterating the questions:

  1. What ways are available (need some sample code) to pass values between Airflow tasks?
  2. What is wrong in the below code? Why is it throwing a jinja template error (cannot render template) '{{ ti.xcom_pull(task_ids="get_fusion_args",key="fusion_args) }}'
Chittu
  • 3
  • 2

1 Answers1

0

I found some answers to the questions:

Q1. What ways are available (need some sample code) to pass values between Airflow tasks? A. Link1 and Link2 mention how to pull xcom values within BashOperator (i.e., without using a python callable). The same can used with other operators.

Q2. Seems there is nothing wrong in the syntax of ti.xcom_pull(task_ids="get_fusion_args",key="fusion_args). What is wrong is that the pull returns a LazyXComAccess([1 items]) which is not getting rendered. I found this by printing the value in the earlier step which pushes the xcom. If the push has been a proper value, then the pull step works fine.

Chittu
  • 3
  • 2