5

I've been trying to remove the new line and carriage return indicators from my JSON output using this answer.

I've also tried running this based on another answer I've seen on Stack Overflow, but it still does not appear to be working:

 sed -E ':a;N;$!ba;s/\r{0,1}\n/\\n/g' text.json > text_clean.json

My current .sh script looks like this:

getPage() {
curl --fail -X GET 'https://api.test.com/v1/marx?page=1&pageSize=1000&sort=desc' \
  -H 'Authorization: Bearer xxxxxxx' \
  -H 'cache-control: no-cache'  
}

getPage \
| jq -c '.data[]' \
  > text.json

What am I missing here?

If it helps, an example of the string containing it takes on many different forms in the output, but here's a good one:

Ok! Thank you so much.\r\nBest,\r\nClaire\r\n\r\n

Emma
  • 27,428
  • 11
  • 44
  • 69
  • @Emma I have tried applying that methodology and it still does not work :( –  Feb 21 '19 at 03:14
  • @Emma I will run `sed -i 's/\r//g' text.json` and it won't really do anything –  Feb 21 '19 at 03:25
  • 1
    @gooponyagrinch If you have `perl` installed, you may be able to get the result you expect from `echo "Ok! Thank you so much.\r\nBest,\r\nClaire\r\n\r\n" | perl -pe 's/\r\n//g'` – Jebby Feb 21 '19 at 03:51
  • 2
    @gooponyagrinch Or by using `tr`: `echo "Ok! Thank you so much.\r\nBest,\r\nClaire\r\n\r\n" | tr -d '\r\n'` – Jebby Feb 21 '19 at 03:54
  • It'd be helpful if we actually could see literal inputs. If the newlines you want to remove are non-literal and within JSON strings, I'd be telling `jq` to do the removal. – Charles Duffy Feb 23 '19 at 02:14
  • @Jebby, `tr` operates on individual characters, so if it's parsing your input to remove a literal backslash (followed by a `r` or `n` or not), it'll also remove a literal `n` and a literal `r` on their own. – Charles Duffy Feb 23 '19 at 02:20

3 Answers3

3

If you want to modify JSON, do it in jq.

In this case, that might mean changing:

getPage \
| jq -c '.data[]' \
> text.json

...to...

getPage \
| jq -c '.data[] | sub("\r\n"; " ") | sub("\n"; " ")' \
> text.json
Charles Duffy
  • 280,126
  • 43
  • 390
  • 441
2

I would use tr for this.

In you example, try the following:

 echo "Ok! Thank you so much.\r\nBest,\r\nClaire\r\n\r\n"| tr -d '\012\015'
LincolnP
  • 164
  • 1
  • 9
  • This presumes those are literal control characters; but as I understand it the OP is actually dealing with literal backslash followed by literal `r` or `n` to represent these characters in JSON. – tripleee Apr 15 '19 at 16:58
  • Just used the same example he is using, but still, even if literal or not, it should still work in bash script. – LincolnP Apr 15 '19 at 17:19
0

You can remove the characters by using perl if you have it installed:

echo "Ok! Thank you so much.\r\nBest,\r\nClaire\r\n\r\n" | perl -pe 's/\r\n//g'
Jebby
  • 1,845
  • 1
  • 12
  • 25