Bash variable substitution for jq query always returns null

Question

File export from company system generates json in a structure that I cannot change. When searching through for specific attributes based on variables within bash (v5.0.2), it always returns null when using any element of substitution. If I hard code the value to be searched, it works. However, need to be able to use variables to support logic of code and to not have to hard code specifics, especially given 1000's of records to process.

Using the example extract of json below, and the code further below, this shows what I've tried based on what I can understand from jq manual.

{
  "status": true,
  "payload": {
    "customerData": {
      "101": {
        "title": "A Company",
        "id": "101",
        "storageGroups": "SHIPPING",
        "priority": "4",
        "originalname": "A Previous Company",
        "allowShipping": true,
        "parentCompany": "0",
        "rating": 0,
        "external": false,
        "application": "primary",
        "applicationName": "form27",
        "live": true,
        "logo": "http://url",
        "beta": false,
        "length": null,
        "token": "QSBQcmV2aW91cyBDb21wYW55LzEwMQ==",
        "active": "1"
      }
    }
  }
}

Simple bash script to evidence approaches tested so far

#!/bin/bash
chkCustomer=101
chkFile="json-simple.txt"
if [[ ! -s "${chkFile}" ]] ; then exit 1 ; fi
tokenTry1=$(cat "${chkFile}" | jq -r '.payload.customerData."${chkCustomer}".token')
printf "%s\n" "Try 1 Returned ${tokenTry1}"
tokenTry2=$(cat "${chkFile}" | jq -r '.payload.customerData."$chkCustomer".token')
printf "%s\n" "Try 2 Returned ${tokenTry2}"
tokenTry3=$(cat "${chkFile}" | jq -r --arg CHK ${chkCustomer} '.payload.customerData."$CHK".token')
printf "%s\n" "Try 3 Returned ${tokenTry3}"
tokenTry4=$(cat "${chkFile}" | jq -r --arg CHK ${chkCustomer} '.payload.customerData."env.$CHK".token')
printf "%s\n" "Try 4 Returned ${tokenTry4}"
tokenTry5=$(cat "${chkFile}" | jq -r --arg CHK 101 '.payload.customerData."$CHK".token')
printf "%s\n" "Try 5 Returned ${tokenTry5}"
tokenTry6=$(cat "${chkFile}" | jq -r '.payload.customerData.101.token')
printf "%s\n" "Try 6 Returned ${tokenTry6}"
tokenWorks=$(cat "${chkFile}" | jq -r '.payload.customerData."101".token')
printf "%s\n" "Try 7 (works) Returned ${tokenWorks}"

Only attempt 7 results in a valid response. All rest are either null or a failure.

Output of the script against the json is

Try 1 Returned null
Try 2 Returned null
Try 3 Returned null
Try 4 Returned null
Try 5 Returned null
jq: error: Invalid numeric literal at EOF at line 1, column 5 (while parsing '.101.') at <top-level>, line 1:
.payload.customerData.101.token                     
jq: error: syntax error, unexpected LITERAL, expecting $end (Unix shell quoting issues?) at <top-level>, line 1:
.payload.customerData.101.token                     
jq: 2 compile errors
Try 6 Returned 
Try 7 (works) Returned QSBQcmV2aW91cyBDb21wYW55LzEwMQ==

What I need (or expect) is that I can use a variable and execute a jq query to get a result.

I'm sure I'm missing something mindbogglingly obvious, but I cannot fathom this. Looking at other questions I've found here on SO, none have yet helped.

`'.payload.customerData."${chkCustomer}".token'` should be `'.payload.customerData.'"${chkCustomer}"'.token'` — oguz ismail, Mar 30 '19 at 16:02
Thanks, but that results in error of `jq: error: Invalid numeric literal at EOF at line 1, column 5 (while parsing '.101.') at , line 1: .payload.customerData.101.token jq: error: syntax error, unexpected LITERAL, expecting $end (Unix shell quoting issues?) at , line 1: .payload.customerData.101.token jq: 2 compile errors ` — , Mar 30 '19 at 16:17
bash quoting seems to be the issue. Escaping the variable within the new ' ' works. `jq -r '.payload.customerData.'\""${chkCustomer}"\"'.token'` results in token being found — , Mar 30 '19 at 16:24
Resolved. Thanks to @oguzismail for the significant pointer to the correct syntax — , Mar 30 '19 at 16:29
@oguzismail, ...that's a correction from the *bash* end, but it's not at all good practice in `jq`. The same general guidelines apply as if the OP were using awk -- data should always be passed out-of-band from code (using `-v` in awk, or `--arg` in jq). — Charles Duffy, Mar 30 '19 at 17:29

Charles Duffy · Accepted Answer · 2019-03-30T17:50:13.320

The safe way to do this is to pass your variables out-of-band from your code. That is:

#!/bin/bash
chkCustomer=101
chkFile="json-simple.txt"

companyToken=$(jq -r --arg chkCustomer "$chkCustomer" \
               '.payload.customerData[$chkCustomer].token' "$chkFile")

printf "%s\n" "Company token for ${chkCustomer} is ${companyToken}."

The $chkCustomer in the single-quoted string is left alone by the shell, and instead interpreted by jq itself to refer to the string value passed in via --arg.

This prevents a value of chkCustomer that contains jq code from being able to generate output that doesn't actually conform to the input file's contents, or -- in the future, when jq adds I/O primitives -- one that performs file I/O and changes contents of your disk.

score 1 · Answer 2 · 2019-03-30T17:02:13.227

1

Using bash variable within additional single quotes to split the command, and then escaping double-quotes for the actual variable substitution, resolves this.

Example of working code, using same json as original question.

#!/bin/bash
chkCustomer=101
chkFile="json-simple.txt"
companyToken=$(jq -r '.payload.customerData.'\"${chkCustomer}\"'.token' "${chkFile}")
printf "%s\n" "Company token for ${chkCustomer} is ${companyToken}."

This method works whether bash variable is itself quoted or not. Without the escape of double-quotes it fails.

edited Mar 30 '19 at 17:02

answered Mar 30 '19 at 16:44

You will want to fix the [useless use of `cat`](/questions/11710552/useless-use-of-cat) too. – tripleee Mar 30 '19 at 16:50
agreed, thanks. Have edited answer to remove unnecessary use of cat – Mar 30 '19 at 17:02
Don't ever substitute variables in generating code (in *any* language; it's just as unsafe to perform literal substitutions into bash/awk/perl/python code). Future versions of jq will have IO support, making this a security risk: Anyone who can set a `chkCustomer` value when you write your code this way will be able to write files on disk. – Charles Duffy Mar 30 '19 at 17:24
@mason362, btw, the claim that it "works whether the bash variable is itself quoted or not" is not quite true -- if your variable has spaces in it, leaving out syntactic quotes will cause your JQ code to be split into two separate arguments, one before the space and one after. (Or if the `nullglob` shell option were set and the value had a `*` in it, it would cause the code as a whole to not be passed to jq at all, since you presumably don't have any filenames that start with `.payload.customerdata`). – Charles Duffy Mar 30 '19 at 17:27
my statement was related to just the example as posted, namely `chkCustomer=101` being quoted or not. Anything that would normally need to be quoted due to spaces or other is I agree clearly required. – Mar 30 '19 at 21:39

Bash variable substitution for jq query always returns null

2 Answers2