Bash: Ignore key value pairs from a JSON that failed to parse using jq

Question

I'm writing a bash script to read a JSON file and export the key-value pairs as environment variables. Though I could extract the key-value pairs, I'm struggling to skip those entries that failed to parse by jq.

JSON (key3 should fail to parse)

{
 "KEY1":"ABC",
 "KEY2":"XYZ",
 "KEY3":"---ABC---\n
dskfjlksfj"

}

Here is what I tried

for pair in $(cat test.json  | jq -r -R  '. as $line | try fromjson catch $line | to_entries | map("\(.key)=\(.value)") | .[]' ); do
    echo $pair
    export $pair
done

And this is the error

jq: error (at <stdin>:1): string ("{") has no keys
jq: error (at <stdin>:2): string ("  \"key1...) has no keys

My code is based on these posts:

What does your input file look like? Why would you use the `-R` option when you already have JSON? — chepner, Feb 02 '23 at 19:50
Is there really a literal newline following `\n`? That means you don't have valid JSON in the first place, which does complicate matters. Otherwise, `jq -r 'to_entries[] | "\(.key)=\(.value|@sh)"' test.json` might suffice. — chepner, Feb 02 '23 at 20:24
@chepner I do have to deal with `\n` and other invalid chars. The goal is to skip the entries that fails to parse — Abhishek Chandran, Feb 02 '23 at 20:29
The better (though not always feasible) way to deal with this is to report the invalid JSON as a bug and get *valid* JSON to parse. — chepner, Feb 02 '23 at 20:30
@chepner Yeah that would have been ideal. But it's not an option for me — Abhishek Chandran, Feb 02 '23 at 20:40
Let us [continue this discussion in chat](https://chat.stackoverflow.com/rooms/251575/discussion-between-abhishek-chandran-and-chepner). — Abhishek Chandran, Feb 02 '23 at 22:17

peak · Accepted Answer · 2023-02-02T21:21:53.257

1

Here's a response to the revised question. Unfortunately, it will only be useful in certain limited cases, not including the example you give. (Basically, it depends on jq's parser being able to recover before the end of file.)

while read -r line ; do 
    echo export "$line"
done < <(< test.json jq -rn '
  def do:
    try inputs catch null
    | objects
    | to_entries[]
    | "\(.key)=\"\(.value|@sh)\"" ;
  recurse(do) | select(.)
')

Note that further refinements may be warranted, especially if there is potentially something fishy about the key names being used as shell variable names.

edited Feb 02 '23 at 21:21

answered Feb 02 '23 at 20:41

peak

105,803
17
152
177

This one is throwing the error `KEY1:Unbound Variable` when I try to do `echo ${KEY1}`. Not sure if it makes any difference - I have added `cat` in front of `test.json` since it throws a permission denied error without it. – Abhishek Chandran Feb 02 '23 at 20:50
Sorry, I had meant to write `< test.json jq ...` – peak Feb 02 '23 at 20:53
Still the same error – Abhishek Chandran Feb 02 '23 at 21:03
@AbhishekChandran You might be assessing a different source of error. You sure `test.json` is readable? Also you did remove the `echo` command no? – konsolebox Feb 02 '23 at 21:18
1

@peak I think you have to separate the echo and export part to align with OP's example. – konsolebox Feb 02 '23 at 21:20
Yeah, it was the `echo` command. It is working now. – Abhishek Chandran Feb 02 '23 at 21:21

peak · Answer 2 · 2023-02-02T20:43:33.703

0

[Note: this response was made to the original question, which has since been changed. The response essentially assumes the input consists of JSONLines interspersed with other lines.)

Since the goal seems to be to ignore lines that don't have valid key-value pairs, you can simply use catch empty:

while read -r line ; do 
    echo export "$line"
done < <(test.json jq -r -R  '
  try fromjson catch empty
  | objects
  | to_entries[]
  | "\(.key)=\"\(.value|@sh)\"" 
')

Note also the use of @sh and of the shell's read, and the fact that .value (in jq) and $line (in the shell) are both quoted. These are all important for robustness, though further refinements might still be necessary for additional robustness.

edited Feb 02 '23 at 20:43

answered Feb 02 '23 at 19:45

peak

105,803
17
152
177

Note that this won't actually export anything to the present environment because `while read` is running in a subshell. You'd need something more like `while read -r line ; do export "$line" ; done < <( < test.json jq -r -R '...' )` – tjm3772 Feb 02 '23 at 20:23
BTW, as a general rule, it's better practice to separate the assignment and the export into two different operations; considerably fewer moving parts that way. `printf -v "${line%%=*}" %s "${line#*=}" || continue; export "${line%%=*}"`, f/e – Charles Duffy Feb 02 '23 at 20:36
@tjm3772 - My original idea was that the export command texts would have been reviewed and then later submitted for execution, but I've adopted your suggestion. Tx. – peak Feb 02 '23 at 20:50

score 0 · Answer 3 · answered Feb 02 '23 at 22:02

Perhaps there is an algorithm that will repair the broken JSON produced by the upstream system. If not, the following is a horrible but possibly useful "hack" that will at least capture KEY1 and KEY2 in the example in the Q:

jq -Rr '
   capture("\"(?<key>[^\"]*)\"[ \t]*:[ \t]*(?<value>[^}]+)") 
   | (.value |= sub("[ \t]+$"; "") )  # trailing whitespace
   | if .value|test("^\".*\"") then .value |= sub("\"[ \t]*[,}[ \t]*$"; "\"") else . end
   | select(.value | test("^\".*\"$") or (contains("\"")|not) )  # a string or not a string
   | "\(.key)=\(.value|@sh)" 
'

score 0 · Answer 4 · answered Feb 02 '23 at 22:08

0

The broken JSON in the example could be repaired in a number of ways, e.g.:

sed '/\\n$/{N; s/\\n\n/\\n/;}'

produces:

{
 "KEY1":"ABC",
 "KEY2":"XYZ",
 "KEY3":"---ABC---\ndskfjlksfj"

}

At least that's JSON :-)

answered Feb 02 '23 at 22:08

peak

105,803
17
152
177

Bash: Ignore key value pairs from a JSON that failed to parse using jq

4 Answers4