37

I have a function like:

getServers() {
    curl -s -X GET ...
}

The output is a complex JSON string. I can parse the JSON like this:

serverId=$(getServers | jq -r ".[] | select(whatever...)")
echo $serverId

But if I store the output of the function call in a variable, I get an error:

servers=$(getServers)
echo $servers | jq .
# jq can not process this
# parse error: Invalid string: control characters from U+0000 through             
echo "$servers" | jq .
# does not work either

U+001F must be escaped at line ...

Even though I $servers contain the same value as function call, jq fails to process it. What's happening here?

$ jq --version
jq-1.5


This is not a duplicate question! This issue roots in existence of \r\n in the string. Adding quotes does not fix the problem.

Answer

Pipe the string into tr '\r\n' ' ':

servers=$(getServers)
echo $servers | tr '\r\n' ' ' | jq .

This script and its output show how adding " does not fix the problem, but tr does:

#!/bin/env bash

get() {
    x=$'multi\r\nline'
    echo $x
}


g=$(get)

echo 'Without "'
echo $g
echo 'With "'
echo "$g"
echo 'With tr'
echo $g | tr '\r\n' '\n'

Output:

Without "
 line
With "
 line
With tr
multi
 line

I hope community votes to reopen this question.

Marcello Romani
  • 2,967
  • 31
  • 40
hpaknia
  • 2,769
  • 4
  • 34
  • 63
  • Which version of `jq` are you using? – Inian Sep 19 '18 at 07:01
  • 3
    Could you quote the variable and try? `echo "$servers" | jq .` – Inian Sep 19 '18 at 07:04
  • 1
    Unless you have a reproducible example, one can't suggest a good solution – Inian Sep 19 '18 at 07:30
  • Sounds like your ‘servers‘ variable contains a nonprinting character. Running ‘set -x‘ before the relevant code will make it print what it's doing as it goes, including invisibles; this should give you a better idea where the problem is coming from. – Gordon Davisson Sep 19 '18 at 07:41
  • 2
    I have executed same and I didnt get any errors. https://pastebin.com/qJ2DxnBm May be you should provide any sample data of your json. – Raja G Sep 19 '18 at 07:48
  • 1
    `set -x` seems to be a cool command. Now I see my variable contains a lot of `\r\n`. – hpaknia Sep 19 '18 at 07:53
  • @tripleee please read the update. This is not a duplicate. – hpaknia Sep 19 '18 at 08:21
  • 2
    If `jq` was able to read the output from `curl` then that is not the root cause of your problem. I suspect your continued lack of quoting causes the shell to replace some shell metacharacters with control characters. – tripleee Sep 19 '18 at 08:23
  • 5
    I had a this error message in the past. Sometimes indeed using `tr '\r\n' ' '`. However I also encounter another issue from time to time: when I execute the command in Jenkins, sometimes the string is too long. I have to write it in a file then use `jq . file.json` instead of `echo .... | jq .` – DidierH Sep 20 '19 at 08:49
  • 1
    Please place answers in Answer blocks. Later, you can accept your own Answer. Also see [How does accepting an answer work?](https://meta.stackexchange.com/q/5234/173448) – jww Dec 22 '19 at 14:27
  • I updated the question with answer. Please vote to reopen. Then I will add the answer. – hpaknia Jul 02 '20 at 19:13
  • 1
    Thanks this worked for me. I first tried DOS2UNIX to change the file line endings but that did not work. – Wil Sep 16 '20 at 11:23
  • That tr command helps a lot! Thanks! – Joseph Ishak Sep 22 '20 at 22:49
  • Just a summary, the data from getServers is not well-formed json and therefore jq is complaining. This is why quoting won't help. Piping the data through tr helps by "fixing" the characters in the data (but actually changes the meaning somewhat). https://stackoverflow.com/a/27516892/224659 – Johan Bergens Jan 07 '21 at 07:54
  • Is it possible that you've enabled the `xpg_echo` bash option? Check your `shopt` output. If `xpg_echo` is enabled, then `echo "a\nb"` will expand the `\n`; normally only `echo -e "a\nb"` would do so. [`echo` is an unportable nightmare, use `printf` instead.](https://unix.stackexchange.com/questions/65803/why-is-printf-better-than-echo) – Martin von Wittich May 06 '21 at 16:11
  • 1
    Also, use quotes as if your life depends on it. Assignments like `servers=$(getServers)` might be safe without quotes, but `echo $servers` definitely is not (regardless of all the problems introduced by `echo` itself). [It is far easier to use double quotes all the time than to remember when they are needed. They are needed most of the time, so you'll need to learn when they aren't needed, not when they are needed.](https://unix.stackexchange.com/a/68748/32931) – Martin von Wittich May 06 '21 at 16:18
  • 6
    Do not use ```tr``` command to fix this issue! Your error is caused by evaluating ```\n``` in your json (stored in varialbe) to actual newlines. Depending on your use case it may be irrelevant, but in some cases it will lead to removal of perfectly valid encoded newlines from the string. Best solution is to replace ```echo $servers | jq ....``` with: ```printf '%s\n' "$servers"| jq ....``` – pawrog May 29 '21 at 16:37

0 Answers0