Read JSON data in a shell script

Question

In shell I have a requirement wherein I have to read the JSON response which is in the following format:

 { "Messages": [ { "Body": "172.16.1.42|/home/480/1234/5-12-2013/1234.toSort", "ReceiptHandle": "uUk89DYFzt1VAHtMW2iz0VSiDcGHY+H6WtTgcTSgBiFbpFUg5lythf+wQdWluzCoBziie8BiS2GFQVoRjQQfOx3R5jUASxDz7SmoCI5bNPJkWqU8ola+OYBIYNuCP1fYweKl1BOFUF+o2g7xLSIEkrdvLDAhYvHzfPb4QNgOSuN1JGG1GcZehvW3Q/9jq3vjYVIFz3Ho7blCUuWYhGFrpsBn5HWoRYE5VF5Bxc/zO6dPT0n4wRAd3hUEqF3WWeTMlWyTJp1KoMyX7Z8IXH4hKURGjdBQ0PwlSDF2cBYkBUA=", "MD5OfBody": "53e90dc3fa8afa3452c671080569642e", "MessageId": "e93e9238-f9f8-4bf4-bf5b-9a0cae8a0ebc" } ] }

Here I am only concerned with the "Body" property value. I made some unsuccessful attempts like:

 jsawk -a 'return this.Body' 

or

 awk -v k="Body" '{n=split($0,a,","); for (i=1; i<=n; i++) print a[i]} 

But that did not suffice. Can anyone help me with this?

Answer 1

There is jq for parsing json on the command line:

 jq '.Body'

Visit this for jq: https://stedolan.github.io/jq/

Answer 2

tl;dr

$ cat /tmp/so.json | underscore select '.Messages .Body' 
["172.16.1.42|/home/480/1234/5-12-2013/1234.toSort"]

Javascript CLI tools

You can use Javascript CLI tools like

• underscore-cli:
• json:select(): CSS-like selectors for JSON.

Example

Select all name children of a addons:

underscore select ".addons > .name"

The underscore-cli provide others real world examples as well as the json:select() doc.

Answer 3

Similarly using Bash regexp. Shall be able to snatch any key/value pair.

key="Body"
re="\"($key)\": \"([^\"]*)\""

while read -r l; do
    if [[ $l =~ $re ]]; then
        name="${BASH_REMATCH[1]}"
        value="${BASH_REMATCH[2]}"
        echo "$name=$value"
    else
        echo "No match"
    fi
done

Regular expression can be tuned to match multiple spaces/tabs or newline(s). Wouldn't work if value has embedded ". This is an illustration. Better to use some "industrial" parser :)

Answer 4

Here is a crude way to do it: Transform JSON into bash variables to eval them.

This only works for:

• JSON which does not contain nested arrays, and
• JSON from trustworthy sources (else it may confuse your shell script, perhaps it may even be able to harm your system, You have been warned)

Well, yes, it uses PERL to do this job, thanks to CPAN, but is small enough for inclusion directly into a script and hence is quick and easy to debug:

json2bash() {
perl -MJSON -0777 -n -E 'sub J {
my ($p,$v) = @_; my $r = ref $v;
if ($r eq "HASH") { J("${p}_$_", $v->{$_}) for keys %$v; }
elsif ($r eq "ARRAY") { $n = 0; J("$p"."[".$n++."]", $_) foreach @$v; }
else { $v =~ '"s/'/'\\\\''/g"'; $p =~ s/^([^[]*)\[([0-9]*)\](.+)$/$1$3\[$2\]/;
$p =~ tr/-/_/; $p =~ tr/A-Za-z0-9_[]//cd; say "$p='\''$v'\'';"; }
}; J("json", decode_json($_));'
}

use it like eval "$(json2bash <<<'{"a":["b","c"]}')"

Not heavily tested, though. Updates, warnings and more examples see my GIST.

Update

(Unfortunately, following is a link-only-solution, as the C code is far too long to duplicate here.)

For all those, who do not like the above solution, there now is a C program json2sh which (hopefully safely) converts JSON into shell variables. In contrast to the perl snippet, it is able to process any JSON, as long as it is well formed.

Caveats:

• json2sh was not tested much.
• json2sh may create variables, which start with the shellshock pattern () {

I wrote json2sh to be able to post-process .bson with Shell:

bson2json()
{
printf '[';
{ bsondump "$1"; echo "\"END$?\""; } | sed '/^{/s/$/,/';
echo ']';
};

bsons2json()
{
printf '{';
c='';
for a;
do
  printf '%s"%q":' "$c" "$a";
  c=',';
  bson2json "$a";
done;
echo '}';
};

bsons2json */*.bson | json2sh | ..

Explained:

• bson2json dumps a .bson file such, that the records become a JSON array
  • If everything works OK, an END0-Marker is applied, else you will see something like END1.
  • The END-Marker is needed, else empty .bson files would not show up.
• bsons2json dumps a bunch of .bson files as an object, where the output of bson2json is indexed by the filename.

This then is postprocessed by json2sh, such that you can use grep/source/eval/etc. what you need, to bring the values into the shell.

This way you can quickly process the contents of a MongoDB dump on shell level, without need to import it into MongoDB first.