Convert a list of dictionaries with similar/duplicate keys into a single dictionary (values would be list) using reduce method

Question

This question is similar to the post "Python - Convert list of single key dictionaries into a single dictionary", where the assumption is that we guarantee different keys in the list of dictionaries. My question here is, what if we have similar keys and how we can utilize the reduce function.

For example, we have:

lst = [{'1': 'A'}, {'1': 'B'}, {'2': 'C'}, {'2': 'D'}, {'3': 'E'}]

And we want to be:

dict = {'1': ['A', 'B'], '2': ['C', 'D'], '3': ['E']}

In addition, the post How to merge multiple dicts with same key? is similar, except that here we would like to utilize the reduce method.

Please make an attempt, then if you get stuck, come back and ask. — wwii, Jul 07 '17 at 04:12
@wwii I really wanted to know why this comment is ok, while on trying to comment "What have you tried?" you get a message from SO telling that you can't do that. — Vinícius Figueiredo, Jul 07 '17 at 04:14
@wwii Earlier today, I was about to comment "What have you tried?" in a question when the user didn't show an attempt. To my surprise, a dialog box came up saying something along those lines: "Don't comment that. If the user hasn't shown any attempts, maybe the user didn't make any" — Vinícius Figueiredo, Jul 07 '17 at 04:19
@wwii Relax, it's alright. That's what the OP needs to hear. — cs95, Jul 07 '17 at 04:30
@wwii Exactly what coldspeed said, I was just confused by this behaviour from the website. — Vinícius Figueiredo, Jul 07 '17 at 04:31
@ViníciusAguiar The reason you couldn't post that content is because [it's been used to death to link to certain websites](https://meta.stackoverflow.com/questions/251309/comments-cant-contain-that-content-what-have-you-tried). And honestly, it's become pretty rude. If the user hasn't tried anything, inform them that they need to show an effort, and come back if they have a question during the process. — Christian Dean, Jul 07 '17 at 04:45
@wwii except that this question is about using reduce method and its efficiency — otayeby, Jul 07 '17 at 15:05

cs95 · Accepted Answer · 2017-07-07T04:42:38.303

2

Okay, taking inspiration from the linked question, you can do this:

In [12]: from collections import defaultdict
    ...: from functools import reduce

In [13]: lst = [{'1': 'A'}, {'1': 'B'}, {'2': 'C'}, {'2': 'D'}, {'3': 'E'}]

In [14]: def foo(r, d):
    ...:     for k in d:
    ...:         r[k].append(d[k])
    ...:         

In [16]: d = reduce(lambda r, d: foo(r, d) or r, lst, defaultdict(list))

In [17]: d
Out[17]: defaultdict(list, {'1': ['A', 'B'], '2': ['C', 'D'], '3': ['E']})

You need an intermediate function to do the update... I think there's a better way to do this, but this is the crux of it.

Now, if you want a cleaner, more readable way, you can do this:

In [12]: from collections import defaultdict

In [30]: lst = [{'1': 'A'}, {'1': 'B'}, {'2': 'C'}, {'2': 'D'}, {'3': 'E'}]

In [31]: d = defaultdict(list)

In [32]: for i in lst:
    ...:     k, v = list(i.items())[0] # an alternative to the single-iterating inner loop from the previous solution
    ...:     d[k].append(v)
    ...:     

In [33]: d
Out[33]: defaultdict(list, {'1': ['A', 'B'], '2': ['C', 'D'], '3': ['E']})

edited Jul 07 '17 at 04:42

answered Jul 07 '17 at 04:27

cs95

379,657
97
704
746

2

.. Yep, no need to use reduce. – wwii Jul 07 '17 at 04:34
1

Also note you'll have to import `defaultdict()` from `collections` to use your second method. – Christian Dean Jul 07 '17 at 04:42
@ChristianDean Fixed! – cs95 Jul 07 '17 at 04:42
I worked out a solution as shown below, but I think this is a better solution, should I comment my solution under a comment here or keep it separate as it is? – otayeby Jul 07 '17 at 04:46
@tiba Personally, your solution is just a slightly windy version of mine, and since they do the same thing the same way, you would probably delete it. But there's no harm leaving it around either. – cs95 Jul 07 '17 at 04:47
I have another question, would it be faster to use reduce with large dictionaries? – otayeby Jul 07 '17 at 04:47
@tiba Not in my opinion. Lambdas are slow and clunky, and on top of that you're calling a second helper function to do the dirty work. I certainly believe a simple loop (my second solution/Christian Dean's solution) would leave this in the dust. – cs95 Jul 07 '17 at 04:49

otayeby · Answer 2 · 2017-07-07T04:42:32.793

I just came up with a solution using a function with the reduce method.

First we define a function that checks if the key was inserted in dict before or not. If it's a new key we add it to the dict with the value in a list, if it's already there we append the value to the existing list.

def dict_list(each_dict):
    d_key = each_dict.keys()[0]
    d_value = each_dict.values()[0]
    if d_key in return_dict:
        return_dict[d_key].append(d_value)
    else:
        return_dict[d_key] = [d_value]
    return return_dict

Then use this function with reduce in a single line as following:

lst = [{'1': 'A'}, {'1': 'B'}, {'2': 'C'}, {'2': 'D'}, {'3': 'E'}]
return_dict = {}
print reduce(lambda return_dict, each_dict: dict_list(each_dict), lst, {})

score 0 · Answer 3 · answered Jul 07 '17 at 04:39

You can use the setdefault() attribute of dictionary objects:

>>> def combine(dictionaries):
    combined_dict = {}
    for dictionary in dictionaries:
        for key, value in dictionary.items():
            combined_dict.setdefault(key, []).append(value)
    return combined_dict

>>> 
>>> lst = [{'1': 'A'}, {'1': 'B'}, {'2': 'C'}, {'2': 'D'}, {'3': 'E'}]
>>> combine(lst)
{'1': ['A', 'B'], '2': ['C', 'D'], '3': ['E']}
>>>

This basically works by first checking if the key already exists in the new dictionary. If it does, we just append the current value to the same key. If not, we create a new key and append that current value to that.

Convert a list of dictionaries with similar/duplicate keys into a single dictionary (values would be list) using reduce method

3 Answers3