Converting JSON response to key value pair using jq

Question

So, I am getting a response from an API that I am calling in a shell script in the following form

[{"id":100000004,"name":"Customs Clearance Requested"},{"id":100000005,"name":"Customs Cleared"},{"id":100000006,"name":"Cargo Loaded to Vessel"}]

I want to create a map out of it that will help me lookup the id's from a name and use it in the shell script. So something like map["Customs Clearance Requested"] would give me 100000004 which I can use further. Can this be done using jq? I am pretty new to shell scripting and jq and got stuck with above thing

This is very closely related to [converting a json object into a bash associative array](https://stackoverflow.com/questions/26717277/converting-a-json-object-into-a-bash-associative-array), maybe even an outright duplicate (unless you don't know how to use jq to convert your current JSON format into the one that question's answers use as input). — Charles Duffy, Feb 13 '20 at 17:30

Charles Duffy · Accepted Answer · 2020-02-13T17:41:04.933

json='[{"id":100000004,"name":"Customs Clearance Requested"},{"id":100000005,"name":"Customs Cleared"},{"id":100000006,"name":"Cargo Loaded to Vessel"}]'

declare -A map
while IFS= read -r -d '' name && IFS= read -r -d '' value; do
  map[$name]=$value
done < <(jq -j '.[] | "\(.name)\u0000\(.id)\u0000"' <<<"$json")

declare -p map  # demo purposes: print the map we created as output

...emits as output:

declare -A map=(["Cargo Loaded to Vessel"]="100000006" ["Customs Clearance Requested"]="100000004" ["Customs Cleared"]="100000005" )

...which you can query exactly as requested:

$ echo "${map['Cargo Loaded to Vessel']}"
100000006

vdavid · Answer 2 · 2020-02-13T17:37:20.677

You could use the select function, e.g.:

data='[{"id":100000004,"name":"Customs Clearance Requested"},{"id":100000005,"name":"Customs Cleared"},{"id":100000006,"name":"Cargo Loaded to Vessel"}]'
jq 'map(select(.["name"] == "Customs Clearance Requested"))' <<< $data

It will get all elements which name equals "Customs Clearance Requested", e.g.:

[
  {
    "id": 100000004,
    "name": "Customs Clearance Requested"
  }
]

If you want to get the id field:

jq 'map(select(.["name"] == "Customs Clearance Requested")["id"])' <<< $data

This will output:

[
  100000004
]

Please note that it will return an array and not a single element because the search does not know how many results will be found.

If you want to generalize this in a shell function, you could write:

function get_id_from_name
{
  # $1=name to search for
  local filter=$(printf 'map(select(.["name"] == "%s")["id"])' "$1")
  jq "$filter"
}

Then call it like that:

get_id_from_name "Customs Clearance Requested" <<< $data

If your data is stored in a file, you could call it this way:

get_id_from_name "Customs Clearance Requested" < /path/to/file.json

Calling `jq` once per item you want to retrieve seems a little unfortunate, vs converting the whole document to a single bash object all at once. — Charles Duffy, Feb 13 '20 at 17:36
And injecting data into strings parsed as code (in any language) is a bad idea. If you want to pass variables to `jq`, that's what `--arg` is for. Thus, something like: `get_id_from_name() { jq -r --arg name "$1" 'map(select(.["name"] == $name)["id"]) | .[]'; }` — Charles Duffy, Feb 13 '20 at 17:36
@CharlesDuffy I assumed that the OP wanted a single result, but now that you mention it I think you’re right that he needs a map built on the bash side. — vdavid, Feb 13 '20 at 17:43

peak · Answer 3 · 2020-02-13T23:02:24.937

0

The following is very similar to @CharlesDuffy's excellent answer but does not assume that the .name and .id values are NUL-free (i.e., do not have any "\u0000" characters):

declare -A map
while read -r name
do
    name=$(sed -e 's/^"//' -e 's/"$//' <<< "$name")
    read -r id
    map[$name]="$id"
done < <(echo "$json" | jq -c '.[]|.name,.id')

The point is that the -j option is like -r (i.e., produces "raw output"), whereas the -c option produces JSON.

This means that if you don't want the .id values as JSON strings, then the above won't be a solution; also, if the .name values contain double-quotes, then you might want to deal with the occurrences of \".

edited Feb 13 '20 at 23:02

answered Feb 13 '20 at 19:22

peak

105,803
17
152
177

`echo $json` is not particularly safe -- if that JSON contains `"Hello * World"`, then the `*` will be replaced with a list of filenames in the current directory. Honestly, if there's a need to handle content with NULs safely, the better answer is to filter them out in jq -- remember that bash uses C strings and can't represent literal NULs at all. – Charles Duffy Feb 13 '20 at 21:07
...the code here would be doubling up literal backslashes, representing tabs as two-character `\t` sequences, and otherwise not be faithfully representing the JSON values as C strings on the shell side. – Charles Duffy Feb 13 '20 at 21:08
See the last code snippet in https://stackoverflow.com/a/51234929/14122 for an example of an answer being deliberately cautious about NUL handling. – Charles Duffy Feb 13 '20 at 21:08

Converting JSON response to key value pair using jq

3 Answers3