Return a python object from GKEPodOperator

Question

I have an Airflow DAG executing calling a GKEPodOperator where a python script is ran to process and load data to BigQuery. Can the GKEPodOperator return like a python list or dictionary from the executed script back to the DAG so I can make use of that to write custom emails using DAGOperator?

Answer 1

First GKEPodOperator is deprecated. You should use GKEStartPodOperator.

Like other operators you can pass values between tasks with Xcom. Note that the GKEStartPodOperator inherits from KubernetesPodOperator thus the xcom mechanism is different than other operators. It works with launching a sidecar container. You can read more about it with examples here.

Now that you have the desired value stored as XCOM (as string) you want to pull it so you can use it in your custom operator:

1. Airflow >= 2.1.0 (not yet released) There is native support for to convert xcom directly into native python object. See PR. You will need to set render_template_as_native_obj=True. You can read more about it in the docs: Rendering Fields as Native Python Objects

2. Airflow < 2.1.0: In the follow up task you need to pull the xcom value and convert it to whatever Python object you'd like. You can see example for it here.