How to merge complex nested dictionaries

Question

I have two dictionaries:

dict1 = {'cm': {'fill': {'user': {'registration': {'flag': 'f'}}}}}
dict2 = {'cm': {'fill': {'user': {'url': 'www.example.com'}}}}

The output I want:

dict3 = {'cm': {'fill': {'user':{'registration': {'flag': 'f'}}, {'url': 'www.example.com'}}}

Here is what I have tried so far:

dict3 = {**dict1, **dict2} # This does not work. It only gives me a `dict1`.

The problem is that dict1 and dict2 can have many embedded keys.

Any idea how to do this?

The reason your expression `dict3 = {**dict1, **dict2}` does not work is because the key/value pairs in `dict2` will override those in `dict1`. Since they have the same top-level key `cm`, `dict2` is overwritten. They need different top-level keys. Not sure if that's your actual data or just an example. Source: https://stackoverflow.com/a/26853961/7586861 — jarcobi889, Mar 21 '18 at 18:56
Seems to be a potential duplicate of https://stackoverflow.com/questions/7204805/dictionaries-of-dictionaries-merge/7205107#7205107. The `merge` function in the linked answer returns the expected dict on the input you have here. We need more information about expected input though. — dbep, Mar 21 '18 at 19:17

Prune · Answer 1 · 2018-03-21T18:58:34.700

2

Correct: that merge won't do what you want. You have accumulation to do multiple levels down. I suspect that the easiest way for you to get what you want -- merging at unspecified (arbitrary) depth -- is to write a recursive routine to do the merge you want.

def dict_merge(d1, d2):
    for key in d1:
        if key in d2:
            one_merge = dict_merge(d1[key], d2[key])
        else:
            one_merge = d1[key]
    for ... # also pick up keys in d2 that are not in d1.

I'll leave it to you whether to handle this logic with set intersection and difference.

edited Mar 21 '18 at 18:58

answered Mar 21 '18 at 18:55

Prune

76,765
14
60
81

If the structure is consistent, I believe `collections.defaultdict` is a good option. – jpp Mar 21 '18 at 18:56
1

Yup -- We'll wait for OP to sort out whether to rely on that consistency. I went for a more general -- and less obvious -- solution. – Prune Mar 21 '18 at 18:59

jpp · Answer 2 · 2018-03-21T18:59:53.723

If the structure of your dictionaries is consistent, you can use collections.defaultdict with a nested approach.

It's possible to convert the nested defaultdict to regular dict objects. But this may not be necessary for your use case.

from collections import defaultdict

dict1 = {'cm': {'fill': {'user': {'registration': {'flag': 'f'}}}}}
dict2 = {'cm': {'fill': {'user': {'url': 'www.example.com'}}}}

rec_dd = lambda: defaultdict(rec_dd)
d = rec_dd()

for i in [dict1, dict2]:
    d['cm']['fill']['user'].update(i['cm']['fill']['user'])

# defaultdict(<function __main__.<lambda>>,
#             {'cm': defaultdict(<function __main__.<lambda>>,
#                          {'fill': defaultdict(<function __main__.<lambda>>,
#                                       {'user': defaultdict(<function __main__.<lambda>>,
#                                                    {'registration': {'flag': 'f'},
#                                                     'url': 'www.example.com'})})})})

score 0 · Answer 3 · answered Mar 21 '18 at 19:21

You can use recursion while ziping the data:

dict1 = {'cm': {'fill': {'user': {'registration': {'flag': 'f'}}}}}
dict2 = {'cm': {'fill': {'user': {'url': 'www.example.com'}}}}
def update_dict(d1, d2):
   return {a:update_dict(b, d) if a == c and list(b.keys())[0] == list(d.keys())[0] else {**b, **d} for [a, b], [c, d] in zip(d1.items(), d2.items())}

Output:

{'cm': {'fill': {'user': {'registration': {'flag': 'f'}, 'url': 'www.example.com'}}}}

How to merge complex nested dictionaries

3 Answers3