json.decoder.JSONDecodeError when getting string from XCOM

Question

In a task, I serialise a dict (converting a dict to string) and I pushed it to XCOM

result[data] = json.dumps({"agents": ["john.doe@example.com"], "houses": ["jane.doe@example.com"]})

In Airflow's UI it looks good as a string, and the DAG level, I get in value XCOM_DATA = "{{ task_instance.xcom_pull(task_ids='task_name', key='return_value')['data']}}"

But when a k8s pod operator, for some reason it deletes the double quotes(")

KubernetesPodOperator(
        cmds=["python", "src/send_notification.py"],
        arguments=[
            "--data",
            XCOM_DATA,
        ],
        task_id="notify",
        name="notify",
        dag=dag,
        **COMMON_TASKS_ARGS,
    )

UPDATE: This is the command passed to K8S

 ['. /secrets/env; python /home/app/src/send_notification.py '
 '"--pr" '
 '"https://gitlab.com/30675" "--data" '
 '"{"agents": ["john.doe@example.com", '
 '"jane.doe@example.com"]}" '

and when I deserialize it and inside the script, I get this error json.loads(args.data)

obj, end = self.raw_decode(s, idx=_w(s, 0).end())
File "/usr/local/lib/python3.9/json/decoder.py", line 353, in raw_decode
obj, end = self.scan_once(s, idx)
json.decoder.JSONDecodeError: Expecting property name enclosed in double quotes: line 1 column 2 (char 1)

I printed the values and the error occurs for some reason airflow deletes the double quotes to the string and that's why the error occurs. Can someone help me to correct it?

I used jinja methods like safe or escape
I convert to json in Jinja

Can you share the log of the task to see what was sent to eks? The command itself — ozs, Apr 29 '23 at 07:35

score 0 · Answer 1 · answered May 02 '23 at 12:36

0

You are sending string to the cli. that inside this string you need to add double quotes because of json format.

in that case you need to send it with escape literals

"{\"agents\": [\"john.doe@example.com\",\"jane.doe@example.com\"]}"

answered May 02 '23 at 12:36

ozs

3,051
1
10
19

score 0 · Answer 2 · answered May 10 '23 at 15:45

0

I fixed it adding | tojson

XCOM_DATA = " \"{{ task_instance.xcom_pull(task_ids='task_name', key='return_value')['data']| tojson \" }} "

answered May 10 '23 at 15:45

Alexander Montoya Ángel

1
1

Your answer could be improved with additional supporting information. Please [edit] to add further details, such as citations or documentation, so that others can confirm that your answer is correct. You can find more information on how to write good answers [in the help center](/help/how-to-answer). – Community May 10 '23 at 19:21

json.decoder.JSONDecodeError when getting string from XCOM

2 Answers2