Bash Shell script extract Json object

Question

In Bash shell script, I want to extract an object. For example, with following json file, I would like to extract dependencies object and it should return me: "dmg": ">= 0.0.0", "build-essential": ">= 0.0.0", "windows": ">= 0.0.0" in whatever format and how do you do that?

// My data 1.json:

{
    "platforms": {
        "amazon": ">= 0.0.0",
        "arch": ">= 0.0.0",
        "centos": ">= 0.0.0",
        "debian": ">= 0.0.0"
    },
    "dependencies": {
        "dmg": ">= 0.0.0",
        "build-essential": ">= 0.0.0",
        "windows": ">= 0.0.0"
    },
    "recommendations": {}
}

// My data 2.json:

{
    "platforms": {
        "amazon": ">= 0.0.0",
        "arch": ">= 0.0.0",
        "centos": ">= 0.0.0",
        "debian": ">= 0.0.0"
    },
    "recommendations": {},
    "dependencies": {
        "dmg": ">= 0.0.0",
        "build-essential": ">= 0.0.0",
        "windows": ">= 0.0.0"
    }
}

// My data 3.json:

{
    "dependencies": {
        "dmg": ">= 0.0.0",
        "build-essential": ">= 0.0.0",
        "windows": ">= 0.0.0"
    },
    "platforms": {
        "amazon": ">= 0.0.0",
        "arch": ">= 0.0.0",
        "centos": ">= 0.0.0",
        "debian": ">= 0.0.0"
    },
    "recommendations": {}
}

// My data 4.json:

{
    "dependencies": {
        "dmg": ">= 0.0.0",
        "build-essential": ">= 0.0.0",
        "windows": ">= 0.0.0"
    }
}

// My data 5.json (compress):

{"dependencies":{"dmg":">= 0.0.0","build-essential":">= 0.0.0","windows":">= 0.0.0"},"platforms":{"amazon":">= 0.0.0","arch":">= 0.0.0","centos":">= 0.0.0","debian":">= 0.0.0"},"recommendations":{}}

Nayeem Zen · Answer 1 · 2014-03-02T07:45:21.543

1

Have you looked at jsawk? I would generally use python for parsing JSON data on UNIX systems, since it usually comes bundled with the OS.

Anyways, you can try this:

awk "/dependencies/,/}/ { print }" test.json | grep ":" | grep -v dependencies

in general, to get text between two patterns/strings:

awk "/Pattern1/,/Pattern2/ { print }" inputFile

and then use grep ":" to get all the lines containing the ':' in the object, and then filter out the object name itself by getting all the subsequent lines not containing the object name

UPDATE: for json not in pretty format

sed "s/[,{}]/&\n/g" prettified.json | awk "/dependencies/,/}/ { print }" | grep ":" | grep -v dependencies | awk '{$1=$1}1'

edited Mar 02 '14 at 07:45

answered Mar 02 '14 at 06:50

Nayeem Zen

2,569
1
17
16

this might be a problem if json file is not pretty format, for example json file is compressed into one single line. Any thought? – Nam Nguyen Mar 02 '14 at 06:56
Updated the regex, now should work for both pretty and ugly formats lol – Nayeem Zen Mar 02 '14 at 07:46
Doesn't work if its a string coming in from a curl response. – JPM Sep 01 '23 at 15:29

score 0 · Answer 2 · answered Mar 02 '14 at 06:37

0

Here is one way with awk:

awk -v RS= -F'},|{' '{print $5}' file | awk 'NF'

$ awk -v RS= -F'},|{' '{print $5}' f | awk 'NF'
    "dmg": ">= 0.0.0",
    "build-essential": ">= 0.0.0",
    "windows": ">= 0.0.0"

answered Mar 02 '14 at 06:37

jaypal singh

74,723
23
102
147

I'm curious, how does it know the block named "dependencies" to extract? I don't see the keyword "dependencies" in your command. – Nam Nguyen Mar 02 '14 at 06:43
@NamNguyen I set the input to paragraph mode `RS=` and the field separator to `},` or `{`. Once the file is split I just pick the field you seek by stating the `$5`. That is the field holding value you need. The last pipe is to remove blank lines. – jaypal singh Mar 02 '14 at 06:45
it's very sort and I'm having hard time to understand your solution. I'm still reading your commend and learn from pro :) . Kinna like your solution :) – Nam Nguyen Mar 02 '14 at 06:48
@NamNguyen Thanks `:)`. It's pretty simple. Just look at your input and count the fields by splitting them at every `},` or `{`. You'll see your block is the `5th` field. – jaypal singh Mar 02 '14 at 06:49

Red Cricket · Answer 3 · 2014-03-02T07:25:39.850

0

$ $ tr -d '\n' < myjson.json | sed -e's/[}{]//g' | sed -e's/.*dependencies\":\(.*\)\s*,.*/\1/g' | sed -e's/^ *//g' | sed -e's/, */, /g'
"dmg": ">= 0.0.0", "build-essential": ">= 0.0.0", "windows": ">= 0.0.0"

edited Mar 02 '14 at 07:25

answered Mar 02 '14 at 06:42

Red Cricket

9,762
21
81
166

look like this solution does not work well for some of my cases. – Nam Nguyen Mar 02 '14 at 07:00
Sorry I didn't know about your cases :) – Red Cricket Mar 02 '14 at 07:02
for example, my case is json file is compressed such as all in one line. But I still appreciate for your input. – Nam Nguyen Mar 02 '14 at 07:05
that's what the `tr -d '\n' < myjson.json` does. – Red Cricket Mar 02 '14 at 07:08
for example, this would fail for this test case: { "b": { }, "dependencies": { "dmg": ">= 0.0.0", "build-essential": ">= 0.0.0", "windows": ">= 0.0.0" }, "a": { } } – Nam Nguyen Mar 02 '14 at 07:14
I could not give all the cases in here so I created gist and this gist is the real file that would not work for your case. https://gist.github.com/gdbtek/9303222 Almost :) – Nam Nguyen Mar 02 '14 at 07:30
If there are going to be many unexpected cases you probably want to look into `jsawk` or using python of perl instead of shell. – Red Cricket Mar 02 '14 at 07:39

axiom · Accepted Answer · 2014-03-02T08:13:42.297

-1

sed -n '/dependencies/, /}/ p' t|grep '>='

How this works :

First get the text between dependencies block, and then extract the dependencies.

Note that this method is independent of where in the text the dependency block is located. As long as it is present, you'll get the answer.

aman@apollo:~$ sed -n '/dependencies/, /\}/ p' t|grep '>='
        "dmg": ">= 0.0.0",
        "build-essential": ">= 0.0.0",
        "windows": ">= 0.0.0"

Use sed -n '/dependencies/, /}/ p' t|grep '.*='

If there can be symbols like ~=, = in the dependency block (and not just >=).

Compressed version For the compressed version, you can first "decompress" (insert newlines) the file and then apply the same transformation.

sed -e 's/:{/:{\n/g'  -e  's/},/\n},\n/g' d5|sed -n '/dependencies/, /}/ p'|grep '>='

The original solution will work for all the 4 other files.

edited Mar 02 '14 at 08:13

answered Mar 02 '14 at 06:32

axiom

8,765
3
36
38

would be safer this way? sed -n '/dependencies/, /}/ p' metadata.json | grep -v "[{}]" – Nam Nguyen Mar 02 '14 at 08:00
do you know the answer for this question? http://stackoverflow.com/questions/22316714/bash-shell-extract-object-from-json-file – Nam Nguyen Mar 11 '14 at 05:01
How is sed reading the file here? makes no sense to me. – JPM Sep 01 '23 at 15:23

Bash Shell script extract Json object

4 Answers4

Linked

Related