adding item to json using bash variable with jq

Question

I am trying to use bash to create a string variable that will be used in jq to add new elements to the json file. But it's escaping my double quote. See below for "wrong output" produced. I am expecting the output shown below "expected output". What is the correct way to add more fields to filename with bash variable?

My input json file (input.json):

{
  "##_Comment1": "Inputs",
  "filename": [
    "file1",
    "file2",
    "file3",
    "file4"
  ]
}

My bash script:

#!/bin/bash
update_list='"abc","efg"'

cat input.json | jq --arg args "$update_list" '.["filename"] += [$args]'

wrong output:

{
  "##_Comment1": "Inputs",
  "filename": [
    "file1",
    "file2",
    "file3",
    "file4",
    "\"abc\",\"efg\""
  ]
}

correct output:

{
  "##_Comment1": "Inputs",
  "filename": [
    "file1",
    "file2",
    "file3",
    "file4",
    "abc",
    "efg"
  ]
}

Not familiar with jq, just a shot in the dark. The problem seems to be that it reads the update_list as string and not as array so maybe `update_list=("abc","efg")` and pass `update_list` to `jq` without double quotes — Theofanis, Jul 30 '18 at 13:51
@Theofanis, that's not a 2-item bash array, because `,` does not separate array items in bash -- `update_list=( abc efg )` would be a native bash array. That said, it would be saner to pass the list as a JSON string itself: `update_list='["abc","efg"]'` — Charles Duffy, Jul 30 '18 at 17:09

Attie · Accepted Answer · 2018-07-30T14:05:36.560

Unwind your situation a bit:

$ jq --arg args '"abc","efg"' '.["filename"] += [$args]' <<< '{"filename":[]}'
{
  "filename": [
    "\"abc\",\"efg\""
  ]
}

Here, we are effectively assigning args to a string in the jq engine:

args = "\"abc\",\"efg\""

If you want set args to a list/array, then you'll need to take another approach.

You could either format a JSON argument and use --argjson:

$ jq --argjson args '["abc","efg"]' '.["filename"] += $args' <<< '{"filename":[]}'
{
  "filename": [
    "abc",
    "efg"
  ]
}

Or you could make update_list into an array, and loop:

update_list=()
update_list+=("abc")
update_list+=("efg")

echo '{"filename":[]}' > test

for i in "${update_list[@]}"; do
    jq --arg update_item "${i}" '.["filename"] += [ $update_item ]' < test \
        | sponge test
done

$ cat test
{
  "filename": [
    "abc",
    "efg"
  ]
}

For the last case, I would consider `--slurpfile` to read the input file as a variable, and `printf '%s\n' "${update_list[@]}"` to serialize the list into a line-oriented string (if we don't need to worry about list elements containing literal newlines) which can be fed on stdin; that way, `jq -Rn` will be able to get that list as `inputs`. — Charles Duffy, Jul 30 '18 at 17:11

peak · Answer 2 · 2018-07-31T00:37:59.553

In the original question, update_list is a comma-separated listing of quoted strings, and the following solution would work if these strings are also JSON strings:

jq --arg args "$update_list" '
  .["filename"] += ($args|split(",")|map(fromjson))'

update_list variants

If update_list can be made available as a JSON array, then @Attie's first solution would be the way to go.

However if update_list is a bash array, the solution involving one call to jq per array element is unnecessarily (and might be embarrassingly) inefficient; the suggested solution might also create problems because the update is not atomic. There are far better alternatives. For example, the jq FAQ mentions a technique which, when applied to the present problem, yields the following solution:

jq --argjson args "$(printf '%s\n' "${update_list[@]}" | jq -nR '[inputs]')" '
 .["filename"] += $args'

Or for robustness, one could use NUL as the delimiter:

jq --argjson args "$(printf '%s\0' "${update_list[@]}" | jq -sR 'split("\u0000")')" '
 .["filename"] += $args'

See also jq convert bash array to json array and insert to file

adding item to json using bash variable with jq

2 Answers2

update_list variants