How to merge json objects containing arrays using python?

Question

I have two json files which contain all kinds of levels of properties. I want to write a python script that will replace existing properties and add missing ones, but keep all the other ones in place.

In my attempts until now the entire "configurations" array of the original file is overwritten, including all properties. All examples I could find show merge for objects without arrays. Any help would be appreciated.

Original:

{
  "configurations": [
    {
      "this-needs-to-stay": {
        "properties": {
          "some_property": "EXISTING"
        }
      }
    },
    {
      "this-needs-to-be-updated": {
        "properties": {
          "this.would.stay": "EXISTING",
          "this.wont.be.overwritten": "EXISTING"
        }
      }
    }
  ],
  "other-values-1": [
    {
      "components": [
        {
          "name": "EXISTING"
        }
      ],
      "name": "somename"
    }
  ],
  "other-values-2": {
    "randomProperties": {
      "type": "random"
    },
    "and_so_on": "you_get_the_point"
  }
}

Additional data that should be added to original:

{
  "configurations" : [
    {
      "this-would-be-added": {
        "properties": {
          "some-property": "ADDED"
        }
      }
    },
    {
      "this-needs-to-be-updated": {
        "properties": {
          "this.would.stay": "CHANGED",
          "this.would.be.added": "ADDED"
        }
      }
    }
  ]
}

Result is a merging of the two on the property level:

{
  "configurations": [
    {
      "this-would-be-added": {
        "properties": {
          "some-property": "ADDED"
        }
      }
    },
    {
      "this-needs-to-stay": {
        "properties": {
          "some_property": "EXISTING"
        }
      }
    },
    {
      "this-needs-to-be-updated": {
        "properties": {
          "this.would.stay": "CHANGED",
          "this.would.be.added": "ADDED"
          "this.wont.be.overwritten": "EXISTING"
        }
      }
    }
  ],
  "other-values-1": [
    {
      "components": [
        {
          "name": "EXISTING"
        }
      ],
      "name": "somename"
    }
  ],
  "other-values-2": {
    "randomProperties": {
      "type": "random"
    },
    "and_so_on": "you_get_the_point"
  }
}

Answer 1

Using funcy.merge:

from funcy import merge

x, y = map(lambda d: {hash(frozenset(c.keys())):c for c in d}, (a['configurations'], b['configurations']))
merged = list(merge(x, y).values())

print(json.dumps(merged, indent=4))

Result:

[
    {
        "this-needs-to-stay": {
            "properties": {
                "some_property": "EXISTING"
            }
        }
    },
    {
        "this-needs-to-be-updated": {
            "properties": {
                "this.would.stay": "CHANGED",
                "this.would.be.added": "ADDED"
            }
        }
    },
    {
        "this-would-be-added": {
            "properties": {
                "some-property": "ADDED"
            }
        }
    }
]   

Answer 2

In the items of configurations in you sample data, looks like you are using items' only key as a unique key in the array. Therefore, we can convert the list into a dict by using that unique key.

That is turning [{"ID_1": "VALUE_1"}, {"ID_2": "VALUE_2"}] into {"ID_1": "VALUE_1", "ID_2": "VALUE_2"}

Then, we just want to merge those two dict. Here I use {**a, **b} to merge them. For this part, you can take a look at How to merge two dictionaries in a single expression?

So

{"ID_1": "value_1", "ID_2": "value_2"}

and

{"ID_2": "new_value_2", "ID_3": "new_value_3"}

would be merged as

{"ID_1": "value_1", "ID_2": "new_value_2", "ID_3": "new_value_3"}

Once they are merged, convert the result dict back into list and that's the final result.

[{"ID_1": "value_1"}, {"ID_2": "new_value_2"}, {"ID_3": "new_value_3"}]

Codes:

def list_to_dict(l):
    return {list(item.keys())[0]: list(item.values())[0] for item in l}

def list_item_merge(a, b):
    return [{k: v} for k, v in {**list_to_dict(a), **list_to_dict(b)}.items()]

list_item_merge(original['configurations'], additional['configurations'])

Answer 3

I would suggest reviewing your conf structure. A list of dicts with single key doesn't make sense to me. Why not just use a dict?:

{
  "configurations": {
    "this-needs-to-stay": {
      "properties": {
        "some_property": "EXISTING"
      }
    },
    "this-needs-to-be-updated": {
      "properties": {
        "this.would.stay": "EXISTING",
        "this.wont.be.overwritten": "EXISTING"
      }
    }

  },
  # ...
}

Then you can simply use:

from funcy import merge

conf = base
conf['configurations'] = merge(base['configurations'],
                               new['configurations'])