Select values that match an element in bash array in JQ

Question

I need to run a script where I have an array of ids, and a json that contains elements with these ids and more data. I need to filter/select the values that matches the ones in the array. For some reason, when running directly on bash I got good results with some of the things I've tried, but not when running with a script. Not sure if this may be relevant or I did something wrong when passing from one to the other

For example, if I have a json file like

[
 {
   "id": 1,
   "value": 10
 },   
 {
   "id": 2,
   "value": 100
 },
 {
   "id": 3,
   "value": 5
 },
 {
   "id": 4,
   "value": 17
 },
 {
   "id": 5,
   "value": 84
 }
]

And the following array:

IDS=(1 2 3 4)

I need to retrieve the values 10, 100, 5 and 17.

I've tried some ways:

VALUES=$(jq --argjson IDS "$IDS" '.[] | select( $IDS =~ .id ) | .value' file.json)
VALUES=$(jq --argjson IDS "$IDS" '.[] | select( ${IDS[*]} =~ .id ) | .value' file.json)
# I have certainty that IDS will always have 4 elements, no more no less
VALUES=$(jq --argjson IDS "$IDS" '.[] | select( .id == ${IDS[0]} | .id == ${IDS[1]} | .id == ${IDS[2]} | .id == ${IDS[3]} ) | .value' file.json)

In all cases I'm getting `unexpected INVALID_CHARACTER (Unix shell quoting issues?)

Also, on the last case, when I replaced the ${IDS[n]} for a hardcoded number, it worked fine. Yet, I won't have the same numbers on each run, so I need that parametrized.

Any help would be great!

EDIT

Thanks everyone for the solutions. I kept the one I understood the most for now, but I'm really greatful with all

Note that `$IDS` in bash just becomes `1`; it evaluates to `${IDS[0]}`. I'd use `--arg` to pass in `${IDS[*]}` and then split them into a JSON list. — Charles Duffy, Feb 27 '23 at 14:41
And `${IDS[0]}` isn't valid jq syntax. It's normal/expected you can't use bash syntax in a jq expression. — Charles Duffy, Feb 27 '23 at 14:42
Also, all-caps variables are in a reserved namespace in shell used for system-meaningful variable names; you should use lowercase names for your own variables. — Charles Duffy, Feb 27 '23 at 14:43
Insofar as this is a question about using shell arrays, answers can be found at https://stackoverflow.com/questions/26808855/how-to-format-a-bash-array-as-a-json-array — peak, Feb 27 '23 at 15:16

oguz ismail · Answer 1 · 2023-02-27T18:27:50.493

2

You're looking for something like this:

$ IDS=(1 2 3 4)
$ jq '.[] | select(IN(.id; $ARGS.positional[])) .value' file.json --jsonargs "${IDS[@]}"
10
100
5
17

edited Feb 27 '23 at 18:27

answered Feb 27 '23 at 14:56

oguz ismail

1
16
47
69

1

Instead of passing a string then split it, one can pass individual bash array elements as json arguments: `jq '.[] | select([.id] | index($ARGS.positional[])).value' --jsonargs "${IDS[@]}"` which also saves the number conversion. – Léa Gris Feb 27 '23 at 17:39
@LéaGris I didn't know about --jsonargs, thank you – oguz ismail Feb 27 '23 at 18:28

0stone0 · Accepted Answer · 2023-02-27T15:03:59.383

2

Use --arg and then convert the Bash array to JSON integers:

($IDS | split(" ") | map(tonumber)) as $PIDS

Combine that with select() and index():

jq \
    --arg IDS "${IDS[*]}" \
    '($IDS | split(" ") | map(tonumber)) as $PIDS | 
        .[] | select([.id] | index($PIDS[])).value' \
input

Gives:

edited Feb 27 '23 at 15:03

answered Feb 27 '23 at 14:56

0stone0

34,288
4
39
64

1

You need braces in `${IDS[*]}` – Charles Duffy Feb 27 '23 at 15:00
1

This is doing an O(n) search of the id list for every input item, right? That's fine as long as we know it'll always be short, but wouldn't scale well if used in a place where the list could grow arbitrarily long. – Charles Duffy Feb 27 '23 at 17:46

Charles Duffy · Answer 3 · 2023-02-27T17:47:42.937

2

One approach is to create a map of acceptable IDs and then do lookups within it:

ids=( 1 2 3 4 )

jq --arg idstr "${ids[*]}" '
  ([$idstr | split(" ")[] | {"key": ., "value": true}] | from_entries) as $idmap
  | .[] | select($idmap[.id | tostring]).value'

edited Feb 27 '23 at 17:47

answered Feb 27 '23 at 15:10

Charles Duffy

280,126
43
390
441

score 2 · Answer 4 · answered Feb 27 '23 at 18:09

Alternatively, pass ids individual array elements as arguments to jq using --args "${ids[@]}" and read the values back into a bash array with mapfile:

#!/usr/bin/env bash

ids=( 1 2 3 4 )

mapfile -t values < <(

jq '
  (
    [$ARGS.positional[] | {"key": ., "value": true}] |
    from_entries
  ) as $idmap |
  .[] |
  select($idmap[.id | tostring]).value' \
  input.json \
  --args "${ids[@]}" 
)

# Debug show content of values array
declare -p values

Sample output:

declare -a values=([0]="10" [1]="100" [2]="5" [3]="17")

Select values that match an element in bash array in JQ

4 Answers4