jq merge json via dynamic sub keys

Question

I think I'm a step off from figuring out how to jq reduce via filter a key to another objects sub-key.

I'm trying to combine files (simplified from Elasticsearch's ILM Explain & ILM Policy API responses):

$ echo '{".siem-signals-default": {"modified_date": "siem", "version": 1 }, "kibana-event-log-policy": {"modified_date": "kibana", "version": 1 } }' > ip1.json
$ echo '{"indices": {".siem-signals-default-000001": {"action": "complete", "index": ".siem-signals-default-000001", "policy" : ".siem-signals-default"} } }' > ie1.json

Such that the resulting JSON is:

{
  ".siem-signals-default-000001": {
    "modified_date": "siem",
    "version": 1
    "action": "complete",
    "index": ".siem-signals-default-000001",
    "policy": ".siem-signals-default"
  }
}

Where ie1 is base JSON and for a child-object, its sub-element policy should line up to ip1's key and copy its sub-elements into itself. I've been trying to build off this, this, and this (from StackOverflow, also this, this, this from external sources). I'll list various rabbit hole attempts building off these, but they're all insufficient:

$ ((cat ie1.json | jq '.indices') && cat ip1.json) | jq -s 'map(to_entries)|flatten|from_entries' | jq '. as $v| reduce keys[] as $k({}; if true then .[$k] += $v[$k] else . end)'
{
  ".siem-signals-default": {
    "modified_date": "siem",
    "version": 1
  },
  ".siem-signals-default-000001": {
    "action": "complete",
    "index": ".siem-signals-default-000001",
    "policy": ".siem-signals-default"
  },
  "kibana-event-log-policy": {
    "modified_date": "kibana",
    "version": 1
  }
}

$ jq --slurpfile ip1 ip1.json '.indices as $ie1|$ie1+{ilm: $ip1 }' ie1.json
{
  ".siem-signals-default-000001": {
    "action": "complete",
    "index": ".siem-signals-default-000001",
    "policy": ".siem-signals-default"
  },
  "ilm": [
    {
      ".siem-signals-default": {
        "modified_date": "siem",
        "version": 1
      },
      "kibana-event-log-policy": {
        "modified_date": "kibana",
        "version": 1
      }
    }
  ]
}

I also expected something like this to work, but it compile errors

$ jq -s ip1 ip1.json '. as $ie1|$ie1 + {ilm:(keys[] as $k; $ip1 | select(.policy == $ie1[$k]) | $ie1[$k]  )}' ie1.json
jq: error: ip1/0 is not defined at <top-level>, line 1:
ip1
jq: 1 compile error

From this you can see, I've determined various ways to join the separate files, but though I have code I thought would play into filtering, it's not correct / taking effect. Does anyone have an idea how to get the filter part working? TIA

Those keys don't match, thats intended? – 0stone0 Jan 14 '22 at 17:39 — 0stone0, Jan 14 '22 at 17:39

pmf · Accepted Answer · 2022-01-14T18:18:09.240

This assumes you are trying to combine the .indices object stored in ie1.json with an object within the object stored in ip1.json. As the keys upon to match are different, I further assumed that you want to match the field name from the .indices object, reduced by cutting off everything that comes after the last dash -, to the same key in the object from ip1.json.

To this end, ip1.json is read in from input as $ip (alternatively you can use jq --argfile ip ip1.json for that), then the .indices object is taken from the first input ie1.json and to the inner object accessed via with_entries(.value …) is added the result of a lookup within $ip at the matching and accordingly reduced .key.

jq '
  input as $ip | .indices | with_entries(.value += $ip[.key | sub("-[^-]*$";"")])
' ie1.json ip1.json

{
  ".siem-signals-default-000001": {
    "action": "complete",
    "index": ".siem-signals-default-000001",
    "policy": ".siem-signals-default",
    "modified_date": "siem",
    "version": 1
  }
}

Demo

If instead of the .indices object's inner field nane you want to have the content of field .index as reference (which in your sample data has the same value), you can go with map_values instead of with_entries as you don't need the field's name anymore.

jq '
  input as $ip | .indices | map_values(. += $ip[.index | sub("-[^-]*$";"")])
'ie1.json ip1.json

Demo

Note: I used sub with a regex to manipulate the key name, which you can easily adjust to your liking if in reality it is more complicated. If, however, the pattern is infact as simple as cutting off after the last dash, then using .[:rindex("-")] instead will also get the job done.

Ah, yay! Thank you! This works perfect. I made that way more complicated than it needed to be. Cheers. — Stef Nestor, Jan 14 '22 at 23:55

score 0 · Answer 2 · answered Jan 14 '22 at 23:57

I also received offline feedback of a simple "workable for my use case" but not exact answer:

$ jq '.indices | map(. * input[.policy])' ie1.json ip1.json
[
  {
    "action": "complete",
    "index": ".siem-signals-default-000001",
    "policy": ".siem-signals-default",
    "modified_date": "siem",
    "version": 1
  }
]

Posting in case someone runs into similar, but other answer's better.

jq merge json via dynamic sub keys

2 Answers2