Passing bash variable to jq

Question

I have written a script to retrieve certain value from file.json. It works if I provide the value to jq select, but the variable doesn't seem to work (or I don't know how to use it).

#!/bin/sh

#this works ***
projectID=$(cat file.json | jq -r '.resource[] | select(.username=="myemail@hotmail.com") | .id')
echo "$projectID"

EMAILID=myemail@hotmail.com

#this does not work *** no value is printed
projectID=$(cat file.json | jq -r '.resource[] | select(.username=="$EMAILID") | .id')
echo "$projectID"

A related issue: passing bash variable to jq filter has the slightly different syntax `jq -r --arg var "$var" '.[$var]'` https://stackoverflow.com/questions/34745451/passing-arguments-to-jq-filter — enharmonic, Feb 19 '20 at 18:11
`EMAILID`is between single quotes and does not get expanded. Write it as `'"$EMAILID"'` to take it out of the single quotes. — user1934428, Mar 24 '23 at 08:53

peak · Accepted Answer · 2023-03-04T16:39:37.280

366

Consider also passing in the shell variable (EMAILID) as a jq variable (here also EMAILID, for the sake of illustration):

   projectID=$(jq -r --arg EMAILID "$EMAILID" '
        .resource[]
        | select(.username==$EMAILID) 
        | .id' file.json)

Postscript

For the record, another possibility would be to use jq's env function for accessing environment variables. For example, consider this sequence of bash commands:

EMAILID=foo@bar.com  # not exported
EMAILID="$EMAILID" jq -n 'env.EMAILID'

The output is a JSON string:

"foo@bar.com"

shell arrays

Unfortunately, shell arrays are a different kettle of fish. Here are two SO resources regarding the ingestion of such arrays:

JQ - create JSON array using bash array with space

Convert bash array to json array and insert to file using jq

edited Mar 04 '23 at 16:39

answered Oct 13 '16 at 17:42

peak

105,803
17
152
177

5

This is the only 100%-safe answer; it lets `jq` properly create the filter using the value, rather than using `bash` to create a string that `jq` interprets as a filter. (Consider what happens if the value of `EMAILID` contains a `)`.) – chepner Oct 16 '16 at 02:35
2

My use case, it works! `function n2 { termux-contact-list |jq -r --arg v1 "$1" '.[] | select(.name==$v1)|.number' }` Calling: `n2 Name1` – Timo Aug 19 '18 at 07:12
6

fyi, jq's `env` function is changed between jq 1.4 and 1.5+, so references to `env.EMAILID` (in this answer's example) do not work in jq 1.4, thus recommending using the `--arg EMAILID "$EMAILID"` construct if you need to use jq 1.4 . Hope this helps; just took me a day+ to figure this out for myself ;) – m0j0hn Oct 04 '18 at 20:15
What if I want to pass a comma-separated list? `select(.username==("a","b","c")` works but `select(.username==($user))` does not – Dhawal Feb 13 '19 at 19:59
@Dhawal - If you can't figure it out, it would be better to ask a new SO Q. – peak Feb 13 '19 at 20:46
for me it only worked when using ` --arg VAR="${VAR}" ` and also I'm not using select my use case is different but This may help someone else . ` jq --arg CORE_NAME="$CORE_NAME" '.["${CORE_NAME}"]' ` – Y Melo Apr 12 '19 at 20:03
2

In my case I had to use parenthesis around the variable name, and since it was inside a select I had to escape it too, like this `jq --arg var "${SHELL_VAR}" '.UserPools[] | select(.Name == "\($var)-some-string")'` – Federico May 23 '19 at 02:24
5

Arguments passed using `--arg` will be passed as strings. If you want to pass data of a different JSON type, use `--argjson`, which will parse the string as JSON, e.g., `--argjson myObj '{"a": [1, 2]}'` – BallpointBen Feb 17 '20 at 16:59
no double quotes around the var, in the jq command! i tried a bash var in escaped double quotes before, and it worked, but this way is cleaner and does not even need them. – nymo Jun 07 '21 at 15:51

score 53 · Answer 2 · answered Oct 13 '16 at 17:41

53

I resolved this issue by escaping the inner double quotes

projectID=$(cat file.json | jq -r ".resource[] | select(.username==\"$EMAILID\") | .id")

answered Oct 13 '16 at 17:41

asidd

2,961
2
10
9

1

I am using this because --arg doesn't play nice with `-c` – dimisjim Apr 30 '20 at 12:56

score 37 · Answer 3 · answered May 15 '20 at 07:47

37

Little unrelated but I will still put it here, For other practical purposes shell variables can be used as -

value=10
jq  '."key" = "'"$value"'"' file.json

answered May 15 '20 at 07:47

markroxor

5,928
2
34
43

score 28 · Answer 4 · answered Nov 07 '19 at 11:00

Posting it here as it might help others. In string it might be necessary to pass the quotes to jq. To do the following with jq:

.items[] | select(.name=="string")

in bash you could do

EMAILID=$1
projectID=$(cat file.json | jq -r '.resource[] | select(.username=='\"$EMAILID\"') | .id')

essentially escaping the quotes and passing it on to jq

Gilles Quénot · Answer 5 · 2023-04-19T14:31:48.837

18

It's a quote issue, you need :

projectID=$(
  cat file.json | jq -r ".resource[] | select(.username==\"$EMAILID\") | .id"
)

If you put single quotes to delimit the main string, the shell takes $EMAILID literally.

"Double quote" every literal that contains spaces/metacharacters and every expansion: "$var", "$(command "$var")", "${array[@]}", "a & b". Use 'single quotes' for code or literal $'s: 'Costs $5 US', ssh host 'echo "$HOSTNAME"'. See
http://mywiki.wooledge.org/Quotes
http://mywiki.wooledge.org/Arguments
http://wiki.bash-hackers.org/syntax/words

edited Apr 19 '23 at 14:31

answered Oct 13 '16 at 17:30

Gilles Quénot

173,512
41
224
223

In my case error as well, similar if not using quotes at all: `termux-contact-list |jq -r '.[] | select(.name=='$1')|.number'`. Result: `jq Compile error` – Timo Aug 19 '18 at 07:04
11

I'm not sure why this comment has been voted up so many times; yes, double quotes allows for variable expansion, but `jq` doesn't like single quotes: `jq: error: syntax error, unexpected INVALID_CHARACTER (Unix shell quoting issues?)` – jdelman May 29 '19 at 19:01
1

If your input is a regular json, then its values would be double (not single) quoted. What works: Using double quotes for entire arg of jq and escaping (with `\\`) any double quotes within, like @asid answered. – GuSuku Nov 14 '19 at 17:35
It does NOT work. **If you put single quotes to delimit the main string, the shell takes $EMAILID literally.** Assuming you set $EMAILID="myemail@hotmail.com", the command `cat file.json | jq -r ".resource[] | select(.username=='$EMAILID') | .id"` actually runs as `cat file.json | jq -r ".resource[] | select(.username==myemail@hotmail.com) | .id"` and incur a compile error for `jq`. – axiqia Nov 26 '20 at 12:35
This got me on the way to solving my issue. My final working string UPLOADED_ARNS=$(aws devicefarm list-uploads --arn ${PROJECT_ARN} | \ jq ".uploads[] | {arn: .arn, name: .name} | select (.name | contains(\"$FILE_EXTENSION\"))" | \ jq --raw-output '.arn') – Josh Graham Sep 30 '21 at 18:19

score 12 · Answer 6 · edited May 26 '21 at 08:11

12

Jq now have better way to access environment variables, you can use env.EMAILID:

projectID=$(cat file.json | jq -r ".resource[] | select(.username==env.EMAILID) | .id")

edited May 26 '21 at 08:11

vbence

20,084
9
69
118

answered Apr 01 '20 at 10:01

DukeLion

356
3
4

1

Hey DukeLion, it would be nice to precise the min required version to have this working. Your solution didn't work on jq version 1.6. – tkrishtop Sep 15 '22 at 09:48
I'm using version "jq-1.6" and the above works – magiccrafter May 11 '23 at 14:03

score 7 · Answer 7 · answered Nov 21 '17 at 23:53

Another way to accomplish this is with the jq "--arg" flag. Using the original example:

#!/bin/sh

#this works ***
projectID=$(cat file.json | jq -r '.resource[] | 
select(.username=="myemail@hotmail.com") | .id')
echo "$projectID"

EMAILID=myemail@hotmail.com

# Use --arg to pass the variable to jq. This should work:
projectID=$(cat file.json | jq --arg EMAILID $EMAILID -r '.resource[] 
| select(.username=="$EMAILID") | .id')
echo "$projectID"

See here, which is where I found this solution: https://github.com/stedolan/jq/issues/626

I had to replace `select(.username=="$EMAILID")` with `select(.username==$EMAILID)` to make it work. (I removed the quotes). I'm using __jq__ version 1.6. — Erik Sjölund, Jan 17 '23 at 16:43

score 3 · Answer 8 · answered Jul 19 '19 at 13:02

I know is a bit later to reply, sorry. But that works for me.

export K8S_public_load_balancer_url="$(kubectl get services -n ${TENANT}-production -o wide | grep "ingress-nginx-internal$" | awk '{print $4}')"

And now I am able to fetch and pass the content of the variable to jq

export TF_VAR_public_load_balancer_url="$(aws elbv2 describe-load-balancers --region eu-west-1 | jq -r '.LoadBalancers[] | select (.DNSName == "'$K8S_public_load_balancer_url'") | .LoadBalancerArn')"

In my case I needed to use double quote and quote to access the variable value.

Cheers.

score 1 · Answer 9 · answered Dec 30 '21 at 11:03

I also faced same issue of variable substitution with jq. I found that --arg is the option which must be used with square bracket [] otherwise it won't work.. I am giving you sample example below:

RUNNER_TOKEN=$(aws secretsmanager get-secret-value --secret-id $SECRET_ID | jq '.SecretString|fromjson' | jq --arg kt $SECRET_KEY -r '.[$kt]' | tr -d '"')

score 0 · Answer 10 · answered Oct 15 '21 at 22:21

In case where we want to append some string to the variable value and we are using the escaped double quotes, for example appending .crt to a variable CERT_TYPE; the following should work:

$ CERT_TYPE=client.reader
$ cat certs.json | jq -r ".\"${CERT_TYPE}\".crt" #### This will *not* work #####
$ cat certs.json | jq -r ".\"${CERT_TYPE}.crt\""

score 0 · Answer 11 · answered Aug 01 '23 at 07:24

0

All solution above is failed, I got Success from below using one equal '=' not two '==' in select

shell_script | sed '1d'|  jq --arg VAR ${SHELL_VAR} -cC '.[]|select(.branch = $VAR).branch'

answered Aug 01 '23 at 07:24

OfusJK

676
1
5
13

Passing bash variable to jq

11 Answers11

Postscript

shell arrays

Linked

Related