Nested merge of dicts

Question

I have multiple dictionaries:

a = {
   "project_1": {
       "roles": ["role1"]
   },
   "project_4": ["foo"]
}

b = {
   "project_1": {
       "roles": ["role2"]
   },
   "project_2": {
       "roles": ["role4"]
   }
}
c = {
    "project_5": {
        "roles": ["role5"]
   }
}

How can I merge these dictionaries to get something like the following:

d = {
    "project_1": {
         "roles": ["role1","role2"]
    },
    "project_2": {
         "roles": ["role4"]
    },
    "project_4": ["foo"],
    "project_5": {
         "roles": ["role5"]
    }
}

From the other questions regarding merging of dictionaries, I see that the answers either overwrite the values in a predetermined way, rely on unique keys, or are for a fixed number of dictionaries. I need to retain nested values of similar keys for 2 or more dictionaries.

The constraints of my problem are:

There are a variable number of keys ("project_X") per dictionary
The dicts are not of fixed depth but have a max depth

I would suggest you flatten dictionary using some delimeter on keys ( dot for example ) than merge like you would merge 1d dictionary and than create nested dictionary from merged 1d dict — Yaroslav Surzhikov, Jun 07 '18 at 22:30
https://stackoverflow.com/questions/6027558/flatten-nested-python-dictionaries-compressing-keys — Yaroslav Surzhikov, Jun 07 '18 at 22:31
The duplicate I marked has solutions for combining dict elements in various ways. YOu should be able to adapt this to your purpose with minor changes. — Prune, Jun 07 '18 at 22:50

Ajax1234 · Accepted Answer · 2018-06-07T22:51:10.417

3

You can use itertools.groupby with recursion:

import itertools
def group(_input):
  d = list(itertools.chain(*list(map(lambda x:list(x.items()), _input))))      
  _s = [[a, [c for _, c in b]] for a, b in itertools.groupby(sorted(d, key=lambda x:x[0]), key=lambda x:x[0])]
  return {a:group(b) if all(isinstance(i, dict) for i in b) else list(itertools.chain(*b)) for a, b in _s}

print(group([a, b, c]))

Output:

{'project_1': {'roles': ['role1', 'role2']}, 'project_2': {'roles': ['role4']}, 'project_4': ['foo'], 'project_5': {'roles': ['role5']}}

edited Jun 07 '18 at 22:51

answered Jun 07 '18 at 22:34

Ajax1234

69,937
8
61
102

Works perfectly! That being said, this function is certainly hard to parse since I'm not familiar with itertools. Could you possibly ELI5 what's going on here? Seems like some really useful functionality! – swigganicks Jun 07 '18 at 22:45
2

@swigganicks Sure. In order to combine values of the same key, the input passed to the function must first be converted to a list of lists and then flattened. That way `itertools.groupby` can be applied to the resulting keys. Once the grouping algorithm has run, `group` is called only any substructures consisting of dictionaries. If no dictionaries are found, it is assumed that the `roles` level has been reached, and the recursion stops. – Ajax1234 Jun 07 '18 at 22:50
It works only for strings but not for a number or bool value. The problem is in "list(itertools.chain(*b))" fails with TypeError: 'bool' object is not iterable – hipertracker May 27 '22 at 00:59

Nested merge of dicts

1 Answers1