2

From API I get JSON that has a string fields with multiple lines and different escape characters. I try to both output this JSON and get one of it's values.

I tried different combinations like this:

response=$(curl -s -X 'POST' \
  ".../api/generated_message/action/generate" \
  -H 'accept: application/json' | jq -r .)

echo $response
generated_message_id=$(echo $response | jq -r '.generated_message_id')

However, I always get parse error: Invalid string: control characters from U+0000 through U+001F must be escaped at line 17, column 96. Seems like I need somehow to get an escaped answer from CURL.

Александр М
  • 306
  • 2
  • 10
  • 4
    The problem isn't missing escapes, it's that your `echo` command is mangling the JSON. At the very least you need to double-quote the variable (`echo "$response"`) to avoid word-splitting and wildcard expansion (see ["I just assigned a variable, but `echo $variable` shows something else"](https://stackoverflow.com/questions/29378566)). `echo` itself may also be doing something stupid with the string, so `printf '%s\n' "$response"` would be safer. – Gordon Davisson Aug 28 '23 at 18:46
  • 1
    You're also using "raw output" when you create `response`; depending on the contents of the response, this might turn valid JSON (like `"string"`) into invalid JSON (like `string`). – Benjamin W. Aug 28 '23 at 18:47

2 Answers2

2

The error message is definitely that you feed jq with non-valid JSON.

You have two possible errors (or both):

  1. You convert the curl output to a non-JSON string with jq -r

  2. echo interprets the C-style escape sequences present in the JSON

The following should fix those problems:

response=$(
    curl -s \
         -X 'POST' \
         -H 'accept: application/json' \
         '.../api/generated_message/action/generate'
)

printf '%s\n' "$response" | jq

generated_message_id=$(
    printf '%s\n' "$response" |
    jq -r '.generated_message_id'
)
Fravadona
  • 13,917
  • 1
  • 23
  • 35
  • `Syntax error: Unterminated quoted string` – Александр М Aug 28 '23 at 19:46
  • 4
    @АлександрМ, the code given by the answer here most certainly has balanced quotes. If the result of your `curl` command is invalid JSON because it _doesn't_ have balanced quotes, that's a problem we aren't given enough information to solve; it's your obligation to ensure that we have enough details to see any problem you're reporting ourselves. (Likewise, if the way you're integration this answer into your own code causes a problem, it's not a problem you've given us the essential details to be able to address). – Charles Duffy Aug 28 '23 at 20:04
0

If you don't need $response in the following part of the script, you should do it in a single step :

generated_message_id=$(
    curl -s \
         -X 'POST' \
         -H 'accept: application/json' \
         '.../api/generated_message/action/generate' |
    jq -r '.generated_message_id'
)
Philippe
  • 20,025
  • 2
  • 23
  • 32
  • 1
    It is always a good idea to store the curl response in some file for caching and not overloading the API server with repeated requests with likely same answer. Then provide jq directly with the cache file as last argument. – Léa Gris Aug 28 '23 at 19:46
  • @LéaGris I think you misunderstood. I clearly put the condition that the result of curl is used only once. – Philippe Aug 28 '23 at 20:22
  • The response may not be needed further down the script, but a persistent cache file will allow re-running the same script multiple times on the same response set without needlessly solicitating the API server. – Léa Gris Aug 28 '23 at 20:33
  • @LéaGris Caching across multiple run of the same script is totally a different story. I don't think it's in OP's mind when he assign curl result to response. – Philippe Aug 28 '23 at 21:28
  • True. Anyway I think it is a good and helpful practice. Very often while coding JSON API response processing scripts, especially with complex jq syntax, it takes multiples attempts before it does exactly what it needs to. I have found it important to start caching the API response into a file from the get go, so I can R&D on the best way to process it without consuming my allotted volume/rate on API requests with trials and errors. – Léa Gris Aug 28 '23 at 21:41
  • As you can see this script is used for debugging purposes and I need both the `generated_message_id` to be stored for future use and the `.` to be printed. – Александр М Aug 29 '23 at 11:11