As for as i figured out, x_com
push needs to execute function which requires context parameters.
However, with sparksubmitoperator
, i can only execute python script file, which i cannot pass context parameter.
How can I pass variable from sparksubmitopertor
to other operators?
xcom_push
example:
def xcom_push_test(**context):
xcom_value = "xcom_push_value"
context['task_instance'].xcom_push(key='xcom_push_value', value=xcom_value)
sparksubmitoperator
example:
spark_job = SparkSubmitOperator(
**application="/opt/airflow/dags/airflow_repo.git/x12345/spark_pythonOps.py",**
application_args=["{{ dag_run.conf['parameter_1'] }}", "{{ dag_run.conf['parameter_2'] }}", "{{ dag_run.conf['user_id'] }}"],
task_id="spark_job",
conn_id = "spark-k8s",
)