How to merge a list of dicts, summing values for repeated keys

Question

In this list of dictionaries, I want to sum up the values of matching keys so that the output is as seen below:

dict_x = [{(1, 2): 100}, {(1, 3): 150}, {(1, 3): 150, (3, 4): 150}, {(4, 5): 10, (1, 3): 10, (3, 4): 10},  {(5, 6): 15}]

output:{(1, 2): 100, (1, 3): 310, (3, 4): 160, (4, 5): 10, (5, 6): 15}

I have read through some pages like How can I count the occurrences of a list item in Python? and Count how many times a part of a key appears in a dictionary python while they count the occurrence of the matching elements, they do not quite do a multiple summation of matching elements. Thank you for your time.

The definition of dict_x won't work: it's a list of dictionaries, but the final element - `(5, 6): 15` - is not a dictionary. Should the final element be `{(5, 6): 15}`? — James Buerger, Apr 21 '16 at 03:20

score 2 · Accepted Answer · edited Apr 21 '16 at 21:05

2

Here's a way to do it in one pass with collections.Counter:

>>> from collections import Counter
>>> sum(map(Counter, dict_x), Counter())
Counter({(1, 2): 100, (1, 3): 310, (3, 4): 160, (4, 5): 10, (5, 6): 15})

Previous response:

Maybe not the most efficient way, but for a small list you can do this in two passes:

>>> keys = [k for d in dict_x for k in d]
>>> {k: sum(d.get(k, 0) for d in dict_x) for k in keys}
{(1, 2): 100, (1, 3): 310, (3, 4): 160, (4, 5): 10, (5, 6): 15}

The first line gets all the keys, and the second line sums the results. I'm almost positive there's some more clever way to do this using python builtins... I'll think on it.

edited Apr 21 '16 at 21:05

ursan

2,228
2
14
21

answered Apr 21 '16 at 03:31

jakevdp

77,104
11
125
160

Thank you. Particularly for the efficient version as it was going to be used for a much larger list – Nobi Apr 21 '16 at 03:40
I like the Counter solution a lot! It is a great use of the arithmetic that one can do with Counters and that a lot of people overlook. – ursan Apr 21 '16 at 20:22

score 2 · Answer 2 · answered Apr 21 '16 at 03:33

2

out_dict = {}
for a in dict_x:
        for b in a.keys():
                out_dict[b] = out_dict.get(b, 0) + a[b]

print out_dict

answered Apr 21 '16 at 03:33

Vikas Madhusudana

1,482
1
10
20

2

I would use `for b, v in a.iteritems():` and avoid the extra lookup of `a[b]`. – Mark Reed Apr 21 '16 at 03:35

score 1 · Answer 3 · answered Jul 19 '21 at 08:48

For a small list Counter works fine and looks pythonic but if you have a huge list of dictionaries, Counter grinds to a halt. Here is one possible solution in that case.

%%timeit
from functools import reduce

a = {'a': 2, 'b':3, 'c':4}
b = {'a': 5, 'c':6, 'x':7}

dict_list = [a, b]

def combine_dicts(d1, d2):
  d = d1.copy()
  for word, count in d2.items():
    d[word] = d.get(word,0) + count
  return d  

final_dict = reduce(combine_dicts, dict_list)
# output
1.49 µs ± 19.7 ns per loop (mean ± std. dev. of 7 runs, 1000000 loops each)

And for comparison here is the result using Counter

%%timeit
from collections import Counter

a = {'a': 2, 'b':3, 'c':4}
b = {'a': 5, 'c':6, 'x':7}
dict_list = [a,b]
final_dict = sum(map(Counter, dict_list), Counter())
#output
7.95 µs ± 87.5 ns per loop (mean ± std. dev. of 7 runs, 100000 loops each)

score 0 · Answer 4 · answered Apr 21 '16 at 03:54

Here's one possible solution (foo can further be optimized):

dict_x = [{(1, 2): 100}, {(1, 3): 150}, {(1, 3): 150, (3, 4): 150}, {(4, 5): 10, (1, 3): 10, (3, 4): 10},  {(5, 6): 15}]
new_dict_x = {}

def foo(x):
    global new_dict_x
    for item in x.keys():
        new_dict_x[item] = new_dict_x.get(item, 0) + x[item]

list(map(lambda x: foo(x),dict_x))

print('Input: {}'.format(dict_x))
print('Output: {}'.format(new_dict_x))

Output:

Input: [{(1, 2): 100}, {(1, 3): 150}, {(1, 3): 150, (3, 4): 150}, {(4, 5): 10, (1, 3): 10, (3, 4): 10}, {(5, 6): 15}]
Output: {(1, 2): 100, (4, 5): 10, (5, 6): 15, (1, 3): 310, (3, 4): 160}

How to merge a list of dicts, summing values for repeated keys

4 Answers4