Add new element to existing JSON array with jq

Question

I want to append an element to an array in a JSON file using the jq``add command, but it's not working.

report-2017-01-07.json file:

{  
   "report": "1.0",
   "data": {  
      "date": "2010-01-07",
      "messages": [  
         {  
            "date": "2010-01-07T19:58:42.949Z",
            "xml": "xml_samplesheet_2017_01_07_run_09.xml",
            "status": "OK",
            "message": "metadata loaded into iRODS successfully"
         },
         {  
            "date": "2010-01-07T20:22:46.949Z",
            "xml": "xml_samplesheet_2017_01_07_run_09.xml",
            "status": "NOK",
            "message": "metadata duplicated into iRODS"
         },
         {  
            "date": "2010-01-07T22:11:55.949Z",
            "xml": "xml_samplesheet_2017_01_07_run_09.xml",
            "status": "NOK",
            "message": "metadata was not validated by XSD schema"
         }
      ]
   }
}

I am using this command:

$ cat report-2017-01-07.json 
| jq -s '.data.messages {"date": "2010-01-07T19:55:99.999Z", "xml": "xml_samplesheet_2017_01_07_run_09.xml", "status": "OKKK", "message": "metadata loaded into iRODS successfullyyyyy"}'
jq: error: syntax error, unexpected '{', expecting $end (Unix shell quoting issues?) at <top-level>, line 1:
.data.messages {"date": "2010-01-07T19:55:99.999Z", "xml": "xml_samplesheet_2017_01_07_run_09.xml", "status": "OKKK", "message": "metadata loaded into iRODS successfullyyyyy"}               
jq: 1 compile error

Here's how I want the output to look:

{
    "report": "1.0",
    "data": {
        "date": "2010-01-07",
        "messages": [{
            "date": "2010-01-07T19:58:42.949Z",
            "xml": "xml_samplesheet_2017_01_07_run_09.xml",
            "status": "OK",
            "message": "metadata loaded into iRODS successfully"
        }, {
            "date": "2010-01-07T20:22:46.949Z",
            "xml": "xml_samplesheet_2017_01_07_run_09.xml",
            "status": "NOK",
            "message": "metadata duplicated into iRODS"
        }, {
            "date": "2010-01-07T22:11:55.949Z",
            "xml": "xml_samplesheet_2017_01_07_run_09.xml",
            "status": "NOK",
            "message": "metadata was not validated by XSD schema"
        }, {
            "date": "2010-01-07T19:55:99.999Z",
            "xml": "xml_samplesheet_2017_01_07_run_09.xml",
            "status": "OKKKKKKK",
            "message": "metadata loaded into iRODS successfullyyyyy"
        }]
    }
}

Don't know this well but is it not an issue that .data.messages is an arrray? So you need something like .data.messages.[] — Vanquished Wombat, Feb 15 '17 at 09:42
Ok - read the docs and installed jq (on have Windows here) and tried - get same error as you. Looks like Windows shell has issues with the way it feeds double quotes into the stream which trips up jq. Could not get anything to work so no answer to your question but you might want to look at escaping the double quotes in the jq command. So `"date"` would become `\"date\"` etc. — Vanquished Wombat, Feb 15 '17 at 10:11
if I use this command I can update all the date attributes. `jq '.data.messages[].date = "2010-01-07T99:99:99.999Z"' report-2017-01-07.json` but I cannot add yet. — Felipe, Feb 15 '17 at 10:41

score 169 · Accepted Answer · edited Dec 13 '22 at 13:46

The |= .+ part in the filter adds a new element to the existing array. You can use jq with filter like:

jq '.data.messages[3] |= . + {
      "date": "2010-01-07T19:55:99.999Z", 
      "xml": "xml_samplesheet_2017_01_07_run_09.xml", 
      "status": "OKKK", 
      "message": "metadata loaded into iRODS successfullyyyyy"
}' inputJson

To avoid using the hardcoded length value 3 and dynamically add a new element, use . | length which returns the length, which can be used as the next array index, i.e.,

jq '.data.messages[.data.messages| length] |= . + {
      "date": "2010-01-07T19:55:99.999Z", 
      "xml": "xml_samplesheet_2017_01_07_run_09.xml", 
      "status": "OKKK", 
      "message": "metadata loaded into iRODS successfullyyyyy"
}' inputJson

(or) as per peak's suggestion in the comments, using the += operator alone

jq '.data.messages += [{
     "date": "2010-01-07T19:55:99.999Z",
     "xml": "xml_samplesheet_2017_01_07_run_09.xml", 
     "status": "OKKK", 
     "message": "metadata loaded into iRODS successfullyyyyy"
}]'

which produces the output you need:

{
  "report": "1.0",
  "data": {
    "date": "2010-01-07",
    "messages": [
      {
        "date": "2010-01-07T19:58:42.949Z",
        "xml": "xml_samplesheet_2017_01_07_run_09.xml",
        "status": "OK",
        "message": "metadata loaded into iRODS successfully"
      },
      {
        "date": "2010-01-07T20:22:46.949Z",
        "xml": "xml_samplesheet_2017_01_07_run_09.xml",
        "status": "NOK",
        "message": "metadata duplicated into iRODS"
      },
      {
        "date": "2010-01-07T22:11:55.949Z",
        "xml": "xml_samplesheet_2017_01_07_run_09.xml",
        "status": "NOK",
        "message": "metadata was not validated by XSD schema"
      },
      {
        "date": "2010-01-07T19:55:99.999Z",
        "xml": "xml_samplesheet_2017_01_07_run_09.xml",
        "status": "OKKK",
        "message": "metadata loaded into iRODS successfullyyyyy"
      }
    ]
  }
}

Use jq-play to dry-run your jq-filter and optimize any way you want.

Since the goal is just to append an element, it would be better to use += but it is worth noting that `.data.messages[3] |= . + X` can here be simplified to `.data.messages[3] = X` as the `.` on the RHS is effectively just `null`. — peak, Feb 15 '17 at 19:52
(windows JQ user here): I wonder if it is possible to write this new data to the same (input) file, instead of creating a temporary file which needs to be renamed afterwords.. — script'n'code, Aug 04 '17 at 01:20
@script'n'code - Using `sponge` is probably still the best option if you have it or can install it (it's part of moreutils). — peak, Jan 31 '18 at 22:09
@peak how can I insert that object to the beginning of the array, couldn’t figure out . Thanks — Nic Huang, May 02 '18 at 02:12
@peak how can I specify the order when inserting the element to the array ? — Harry Ma, Nov 06 '19 at 10:01

score 70 · Answer 2 · edited Jul 31 '19 at 08:50

70

Rather than using |=, consider using +=:

.data.messages += [{"date": "2010-01-07T19:55:99.999Z",
   "xml": "xml_samplesheet_2017_01_07_run_09.xml",
   "status": "OKKK", "message": "metadata loaded into iRODS successfullyyyyy"}]

Prepend

On the other hand, if (as @NicHuang asked) you want to add the JSON object to the beginning of the array, you could use the pattern:

 .data.messages |= [ _ ] + .

edited Jul 31 '19 at 08:50

Rob Bednark

25,981
23
80
125

answered Feb 15 '17 at 18:50

peak

105,803
17
152
177

4

Prepend example: `echo '{"arr": [ "data1" ]}' | jq '.arr |= ["data2"] + .'` => `{"arr": ["data2", "data1"]}` – Vitalii Zinchenko Jun 14 '21 at 22:04

score 45 · Answer 3 · answered Aug 17 '19 at 14:30

Summary: ". +" is your saviour

Details:

For adding an entry to a list: You can append [list1] + [list2] (and not [list] + data)

$ echo '[ "data1" ]' | jq '. + [ "data2" ]'
[
  "data1",
  "data2"
]

$ echo '[ {"key1": "value1"} ]' | jq '. + [{"key2": "value2"}]'
[
  {
    "key1": "value1"
  },
  {
    "key2": "value2"
  }
]

For adding a key/value to a dictionary:

$ echo '{"key1": "value1"}' | jq '. + {"key2": "value2"}'
{
  "key1": "value1",
  "key2": "value2"
}

References:

https://gist.github.com/joar/776b7d176196592ed5d8

Add new element to existing JSON array with jq

3 Answers3

Prepend

Linked

Related