Merge multiple nested dictionaries

Question

I have an unknown number of dictionaries with an unknown set of keys, e.g.:

d1 = {'job': {'data': {'id': 'string'}}}
d2 = {'job': {'data': {'title': 'string'}}}
d3 = {'job': {'metadata': {'date': 'string'}}}
d4 = {'user': {'id': 'string'}}

I want one combined dictionary that looks like this:

{
    'job': {
        'data': {
            'id': 'string',
            'title': 'string'
        }
        'metadata': {
            'date': 'string'
        }
    },
    'user': {
        'id': 'string'
    }
}

The built-in update doesn't give me what I want:

>>> combined = {}
>>> combined.update(d1)
>>> combined.update(d2)
>>> combined.update(d3)
>>> combined.update(d4)
>>> combined
{'job': {'metadata': {'date': 'string'}}, 'user': {'id': 'string'}}

This question sounds really close to what I want, but it has the same result:

>>> {**d1, **d2, **d3, **d4}
{'job': {'metadata': {'date': 'string'}}, 'user': {'id': 'string'}}

That question specifically asks for a union, but that doesn't look like a union to me. What am I missing?

If the keys are static, could be as easy as `{**d1['job']['data'], **d2['job']['data']}` to merge the inner dicts — TayTay, Oct 16 '20 at 16:30

Chiheb Nexus · Accepted Answer · 2020-10-16T22:20:58.387

Here is a generic attempt that responds to your question using a recursion, a generator and reduce:

from functools import reduce


def union_nested_dicts(a, b):
    a_keys = list(a.keys()) if isinstance(a, dict) else []
    b_keys = list(b.keys()) if isinstance(b, dict) else []
    for k in set(a_keys + b_keys):
        if not isinstance(a.get(k, {}), dict) or not isinstance(b.get(k, {}), dict):
            if a:
                # Or: yield from a.items()
                yield k, a.get(k, {})
            if b:
                yield from b.items()
        else:
            tmp = dict(union_nested_dicts(a.get(k, {}), b.get(k, {}))) 
            yield k, tmp


a = {'job': {'data': {'id': 'string'}}}
b = {'job': {'data': {'title': 'string'}}}
d3 = {'job': {'metadata': {'date': 'string'}}}
d4 = {'user': {'id': 'string'}}
out = reduce(
    lambda left, right: dict(union_nested_dicts(left, right)),
    [a, b, d3, d4]
)
print(out)

Output:

{
    'job': {
         'data': {'id': 'string', 'title': 'string'},
         'metadata': {'date': 'string'}
     },
     'user': {'id': 'string'}
}

score 1 · Answer 2 · edited Oct 22 '20 at 20:28

1

AFAIU you want to merge dicts with similar keys and update for different ones.

Here's a simple attempt using the new syntax of python 3.9:

def nested_union(a, b):
    new_dict = a.copy()

    for key, value in a.items():
        if key not in b:
            new_dict |= b  # new_dict.update(b)
        elif not isinstance(value, dict):
            new_dict = a | b  # {**a, **b}
        else:
            new_dict[key] = nested_union(value, b[key])

    return new_dict

I suggest you unit test the provided code as my tests were based in your example. A extension with *args would improve readability during usage.

edited Oct 22 '20 at 20:28

Big McLargeHuge

14,841
10
80
108

answered Oct 16 '20 at 17:39

gmacro

397
3
6

Thanks for adding the syntax for earlier versions in comments. – Big McLargeHuge Oct 22 '20 at 20:34
I found some surprising behavior: if the first argument is an empty dictionary, the result is an empty dictionary. Reversing the arguments produces the desired result. – Big McLargeHuge Oct 22 '20 at 21:45
More weirdness - try merging all of these: `[{'a': {'1': 1}}, {'a': {'2': 2}}, {'b': {'1': 1}}, {'b': {'2': 2}}`. Somehow the outer `a` key gets lost. – Big McLargeHuge Oct 22 '20 at 22:35

Merge multiple nested dictionaries

2 Answers2