0

As for as i figured out, x_com push needs to execute function which requires context parameters.

However, with sparksubmitoperator, i can only execute python script file, which i cannot pass context parameter.

How can I pass variable from sparksubmitopertor to other operators?

xcom_push example:

def xcom_push_test(**context):
    xcom_value = "xcom_push_value"
    context['task_instance'].xcom_push(key='xcom_push_value', value=xcom_value)

sparksubmitoperator example:

spark_job = SparkSubmitOperator(  
        **application="/opt/airflow/dags/airflow_repo.git/x12345/spark_pythonOps.py",**
        application_args=["{{ dag_run.conf['parameter_1'] }}", "{{ dag_run.conf['parameter_2'] }}", "{{ dag_run.conf['user_id'] }}"],
        task_id="spark_job", 
        conn_id = "spark-k8s",
    ) 
icyknight
  • 35
  • 6
sko1324
  • 3
  • 2

0 Answers0