Recursively merge dicts so that elements with shared key are combined into a list

Question

I have two dicts that I want to merge:

a = {"name": "john",
     "phone":"123123123",
     "owns": {"cars": "Car 1", "motorbikes": "Motorbike 1"}}

b = {"name": "john",
     "phone":"123",
     "owns": {"cars": "Car 2"}}

If a and b have a common key on the same nesting level, the result should be a list, with both values in it, which is assigned as the value for the shared key.

The result should look like this:

{"name": "john",
 "phone":["123123123","123"],
 "owns": {"cars": ["Car 1", "Car 2"], "motorbikes": "Motorbike 1"}}

Using a.update(b) does not work since it overwrites the shared value of a with the shared value of b, such that result is something like this:

{'name': 'john', 'phone': '123', 'owns': {'cars': 'Car 2'}}

The goal is to merge the dicts without overwriting and to keep all information related to a specific key (in either of the dicts).

Answer 1

With recursion, you can build a dictionary comprehension that accomplishes that.

This solution also takes into account that you might want to later merge more than two dictionaries, flattening the list of values in that case.

def update_merge(d1, d2):
    if isinstance(d1, dict) and isinstance(d2, dict):
        # Unwrap d1 and d2 in new dictionary to keep non-shared keys with **d1, **d2
        # Next unwrap a dict that treats shared keys
        # If two keys have an equal value, we take that value as new value
        # If the values are not equal, we recursively merge them
        return {
            **d1, **d2,
            **{k: d1[k] if d1[k] == d2[k] else update_merge(d1[k], d2[k])
            for k in {*d1} & {*d2}}
        }
    else:
        # This case happens when values are merged
        # It bundle values in a list, making sure
        # to flatten them if they are already lists
        return [
            *(d1 if isinstance(d1, list) else [d1]),
            *(d2 if isinstance(d2, list) else [d2])
        ]

Example:

a = {"name": "john", "phone":"123123123",
     "owns": {"cars": "Car 1", "motorbikes": "Motorbike 1"}}
b = {"name": "john", "phone":"123", "owns": {"cars": "Car 2"}}

update_merge(a, b)
# {'name': 'john',
#  'phone': ['123123123', '123'],
#  'owns': {'cars': ['Car 1', 'Car 2'], 'motorbikes': 'Motorbike 1'}}

Example with more than two objects merged:

a = {"name": "john"}
b = {"name": "jack"}
c = {"name": "joe"}

d = update_merge(a, b)
d = update_merge(d, c)

d # {'name': ['john', 'jack', 'joe']}

Answer 2

You can use itertools.groupby and recursion:

import itertools, sys
a = {"name": "john", "phone":"123123123", "owns": {"cars": "Car 1", "motorbikes": "Motorbike 1"}}
b = {"name": "john", "phone":"123", "owns": {"cars": "Car 2"}}
def condense(r):
  return r[0] if len(set(r)) == 1 else r

def update_dict(c, d):
  _v = {j:[c for _, c in h] for j, h in itertools.groupby(sorted(list(c.items())+list(d.items()), key=lambda x:x[0]), key=lambda x:x[0])}
  return {j:update_dict(*e) if all(isinstance(i, dict) for i in e) else condense(e) for j, e in _v.items()}

print(update_dict(a, b))

Output:

{'name': 'john', 'owns': {'cars': ['Car 1', 'Car 2'], 'motorbikes': 'Motorbike 1'}, 'phone': ['123123123', '123']}

Answer 3

Using sets and things, can also merge any number of dictionaries:

from functools import reduce
import operator

# Usage: merge(a, b, ...)
def merge(*args):
    # Make a copy of the input dicts, can be removed if you don't care about modifying
    # the original dicts.
    args = list(map(dict.copy, args))

    # Dict to store the result.
    out = {}

    for k in reduce(operator.and_, map(dict.keys, args)):  # Python 3 only, see footnotes.
        # Use `.pop()` so that after the all elements of shared keys have been combined,
        # `args` becomes a list of disjoint dicts that we can merge easily.
        vs = [d.pop(k) for d in args]

        if isinstance(vs[0], dict):
            # Recursively merge nested dicts
            common = merge(*vs)
        else:
            # Use a set to collect unique values
            common = set(vs)
            # If only one unique value, store that as is, otherwise use a list
            common = next(iter(common)) if len(common) == 1 else list(common)

        out[k] = common

    # Merge into `out` the rest of the now disjoint dicts
    for arg in args:
        out.update(arg)

    return out

Assuming that each dictionary to be merged have the same "structure", so "owns" can't be a list in a and a dict in b. Each element of the dict needs to be hashable as well since this method uses sets to aggregate unique values.

---

The following, only works in Python 3 since in Python 2, dict.keys() returns a plain old list.

reduce(operator.and_, map(dict.keys, args))

An alternative would be to add an additional map() to convert the lists to sets:

reduce(operator.and_, map(set, map(dict.keys, args)))

Answer 4

Here is a generalized solution to support arbitrary number of arguments:

def _merge_dicts(dict_args):
    if not isinstance(dict_args[0], dict):
        return list(set(dict_args)) if len(set(dict_args)) > 1 else dict_args[0]
    keys = set().union(*dict_args)
    result = {key: 
              _merge_dicts(([d.get(key, None) for d in dict_args if d.get(key, None) is not None])) 
              for key in keys}
    return result
def merge_dicts(*dict_args):
    return _merge_dicts(dict_args)

a = {"name": "john",
     "phone":"123123123",
     "owns": {"cars": "Car 1", "motorbikes": "Motorbike 1"}}

b = {"name": "john",
     "phone":"123",
     "owns": {"cars": "Car 2"}}

merge_dicts(a, b)

yields

{'name': 'john',
 'owns': {'motorbikes': 'Motorbike 1', 'cars': ['Car 2', 'Car 1']},
 'phone': ['123123123', '123']}