How to merge objects with InstanceId unique in bash shell?

Question

I have two json files as below:

I wanna merge objects in tmp1.json and tmp2.json with InstanceId unique value in bash shell.

I have tried jq with argjson option but my jq 1.4 version not support this option. Sorry, I unable update jq to 1.5 version.

#cat tmp1.json
{
  "VolumeId": "vol-046e0be08ac95095a",
  "Instances": [
    {
      "InstanceId": "i-020ce1b2ad08fa6bd"
    }
  ]
}
{
  "VolumeId": "vol-007253a7d24c1c668",
  "Instances": [
    {
      "InstanceId": "i-0c0650c15b099b993"
    }
  ]
}

#cat tmp2.json
{
  "InstanceId": "i-0c0650c15b099b993",
  "InstanceName": "Test1"
}
{
  "InstanceId": "i-020ce1b2ad08fa6bd",
  "InstanceName": "Test"
}

My desired is:

{
      "VolumeId": "vol-046e0be08ac95095a",
      "Instances": [
        {
          "InstanceId": "i-020ce1b2ad08fa6bd"
          "InstanceName": "Test"
        }
      ]
    }
    {
      "VolumeId": "vol-007253a7d24c1c668",
      "Instances": [
        {
          "InstanceId": "i-0c0650c15b099b993"
          "InstanceName": "Test1"
        }
      ]
    }

I was going to suggest a method that required flattening the volumes and then joining them with the instances and then unflattening. Well, see https://stackoverflow.com/questions/39830426/join-two-json-files-based-on-common-key-with-jq-utility-or-alternative-way-from for the join step. It might be possible to use that and do the join without flattening. — Dan D., Dec 12 '18 at 08:30

peak · Accepted Answer · 2018-12-12T09:50:19.377

#!/bin/bash

JQ=jq-1.4

# For ease of understanding, the following is a bit more verbose than
# necessary.  
# One way to get around the constraints of using jq 1.4 is
# to use the "slurp" option so that the contents of the two files can
# be kept separately.

# Note that jq 1.6 includes the following def of INDEX, but we can use it with jq 1.4.

($JQ -s . tmp1.json ; $JQ -s . tmp2.json) | $JQ -s '

def INDEX(stream; idx_expr):
  reduce stream as $row ({};
    .[$row|idx_expr|
      if type != "string" then tojson
      else .
      end] |= $row);

.[0] as $tmp1
| .[1] as $tmp2
| INDEX($tmp2[]; .InstanceId) as $dict
| $tmp1
| map( .Instances |= map(.InstanceName = $dict[.InstanceId].InstanceName))
| .[]
'

Streamlined

INDEX(.[1][]; .InstanceId) as $dict
| .[0][]
| .Instances |= map(.InstanceName = $dict[.InstanceId].InstanceName)

thanks bro. Excellent answer! I am completely convinced – donald.sys Dec 12 '18 at 09:44 — donald.sys, Dec 12 '18 at 09:44

score -1 · Answer 2 · answered Dec 12 '18 at 10:01

minify the two json files

try the following command:

cat tmp2.json|jq -r '"\(.InstanceId) \(.InstanceName)"'|xargs -n2 sh -c 'cat tmp1.json|jq "if .Instances[0].InstanceId==\"$0\" then .Instances[0].InstanceName=\"$1\" else empty end"'

Here is the output:

{
  "VolumeId": "vol-007253a7d24c1c668",
  "Instances": [
    {
      "InstanceId": "i-0c0650c15b099b993",
      "InstanceName": "Test1"
    }
  ]
}
{
  "VolumeId": "vol-046e0be08ac95095a",
  "Instances": [
    {
      "InstanceId": "i-020ce1b2ad08fa6bd",
      "InstanceName": "Test"
    }
  ]
}

How to merge objects with InstanceId unique in bash shell?

2 Answers2

Streamlined