Convert bash array to json array and insert to file using jq

Question

Given a bash array, how to convert it to a JSON array in order to output to a file with jq?

Additionnally: is there a way to keep the server_nohup array unchanged instead of re-writing the whole json file each time?

newArray=(100 200 300)
jq -n --arg newArray $newArray '{
    client_nohup: [ 
        $newArray
    ],
    server_nohup: [

    ]
}' > $projectDir/.watch.json

Current output:

{
"client_nohup": [
    "100"
],
"server_nohup": []
}

Desired output:

{
"client_nohup": [
    100,
    200,
    300
],
"server_nohup": []
}

peak · Accepted Answer · 2018-03-09T16:55:41.290

8

(1) If all the values in newArray are valid as JSON values without spaces, then you could get away with piping the values as a stream, e.g.

newArray=(100 200 300)
echo "${newArray[@]}" |
  jq -s '{client_nohup: ., server_nohup: []}'

(2) Now let's suppose you merely wish to update the "nohup" object in a file, say nohup.json:

{ "client_nohup": [], "server_nohup": [ "keep me" ] }

Since you are using bash, you can then write:

echo "${newArray[@]}" |
  jq -s --argjson nohup "$(cat nohup.json)" '
    . as $newArray | $nohup | .client_nohup = $newArray
  '

Output

(1)

{
  "client_nohup": [
    100,
    200,
    300
   ],
  "server_nohup": []
}

(2)

{
  "client_nohup": [
    100,
    200,
    300
  ],
  "server_nohup": [
    "keep me"
  ]
}

Other cases

Where there's a will, there's a jq way :-)

See for example the accepted answer at How to format a bash array as a JSON array (though this is not a completely generic solution).

For a generic solution, see : How can a variable number of arguments be passed to jq? How can a bash array of values be passed in to jq as a single argument? at the jq FAQ https://github.com/stedolan/jq/wiki/FAQ

Generic Solutions

To be clear, if the array values are known to be valid JSON, there are several good options; if the array values are arbitrary bash strings, then the only efficient, generic way to handle them with jq is by using the -R jq option (e.g. in conjunction with -s), but then the (bash) strings will all be read in as JSON strings, so any intended type information will be lost. (The point here hinges on the technicality that bash strings cannot CONTAIN NUL characters.)

Often, to alleviate the latter concern, one can convert numeric strings to JSON numbers, e.g. using the jq idiom: (tonumber? // .).

edited Mar 09 '18 at 16:55

answered Mar 09 '18 at 00:27

peak

105,803
17
152
177

Thanks for a thorough answer, but does case 1 need the value for key `client_nohup` to be an array nested in another array? – chicks Mar 09 '18 at 01:08
1

It seems I copied-and-pasted the wrong output. Fixed. – peak Mar 09 '18 at 02:12
This would work for simple values, but in general will fail because the array elements could contain newlines or other invalid JSON characters. – chepner Mar 09 '18 at 13:43
@chepner -please note that your point was made in the answer, both at the end and in the final section, which points to techniques which can be applied in other cases. – peak Mar 09 '18 at 14:04
The answer in the FAQ fails for the array `x=(1 $'a\nb' 2)`. – chepner Mar 09 '18 at 14:28
@chepner - Please read the FAQ more carefully. In particular, your case is covered using NUL. – peak Mar 09 '18 at 14:54
No, it isn't. The problem is that `$'a\n'b` is not a valid JSON value (it contains a newline character. If you are sure that each element of the array is already valid JSON, there would be no problem with using a newline to separate the values. – chepner Mar 09 '18 at 15:31
Please read my counterexample more closely; I am using `$'a\nb'` to include a *literal* newline, not `'a\nb'` which contains a digraph that can be interpreted as a newline. `-R` does not help. My point is, indeed, that you cannot encode an *arbitrary* `bash` array, only an array of JSON values. – chepner Mar 09 '18 at 16:20
Ok, my apologies, I did gloss over "are valid as JSON values" in the first sentence. – chepner Mar 09 '18 at 16:32
And my apologies - the FAQ example should also have included the -s option in addition to -R. Fixed. – peak Mar 09 '18 at 16:40
@peak is there any way I could maintain the format instead of becoming a string line. Now I have array=(“xxx”,”yyy”), I want it to insert to file like [xxx,yyy] not simply string xxx yyy – Nic Huang May 04 '18 at 08:10
@NicHuang - Sorry, I do not understand the question. If your question isn't the same as the one here, why not ask yours as a top-level SO question? – peak May 04 '18 at 14:11

chepner · Answer 2 · 2018-03-09T14:40:11.543

-1

In general, the only truly safe way to do this is with multiple invocations of jq, adding each element to the output of the previous command.

arr='[]'  # Empty JSON array
for x in "${newArray[@]}"; do
  arr=$(jq -n --arg x "$x" --argjson arr "$arr" '$arr + [$x]')
done

This ensures that each element x of your bash array is properly encoded prior to adding it to the JSON array.

This is complicated, though, by the fact that bash doesn't not distinguish between numbers and strings. This encodes your array as ["100", "200", "300"], not [100, 200, 300]. In the end, you need to have some awareness of what your array contains, and preprocess it accordingly.

edited Mar 09 '18 at 14:40

answered Mar 09 '18 at 14:23

chepner

497,756
71
530
681

1

It is not the case that ‘the only truly safe way to do this is with multiple invocations of jq’. The jq FAQ shows how this can be avoided using NUL. Search for NUL in the FAQ https://github.com/stedolan/jq/wiki/FAQ – peak Mar 09 '18 at 15:01
One can get around the number/string problem using `tonumber? // .`, though of course that would convert quoted numbers to numbers ... – peak Mar 09 '18 at 15:04
@peak The problem here isn't the separator, but that bash strings aren't necessarily valid JSON strings. It's broken for the same reason that a hard-coded value like `$'"foo\bar"'` isn't valid JSOn. – chepner Mar 09 '18 at 15:24
1

As mentioned elsewhere, you have evidently overlooked the use of the -R option. Of course this forces everything to be a JSON string, and thus your point about the distinction between numbers and strings being lost in a "generic" solution is perfectly valid.0. – peak Mar 09 '18 at 16:09

Convert bash array to json array and insert to file using jq

2 Answers2

Output

Other cases

Generic Solutions

Linked