For a list of lists, merge the dictionaries with collections.defaultdict

Question

This is a simple question, but I'm confused by the behavior of collections.defaultdict. This is to help me learn how this works.

This question is an extrapolation from this useful question: How to merge a list of multiple dictionaries into a dictionary of lists?

Let's now say I have a list of lists of dictionaries. I want to merge the dictionaries as detailed in the above question:

list_of_dictionaries2 = [[{0:3523, 1:3524, 2:3540, 4:3541, 5:3542}, 
    {0:7245, 1:7246, 2:7247, 3:7248, 5:7249, 6:7250},
    {1:20898, 2:20899, 3:20900, 4:20901, 5:20902}], [{0:3, 1:4, 2:5, 3:6}]]

The intended answer is this:

correct2 = [[{0:[3523, 7245], 1:[3524, 7246, 20898], 2:[3540, 7247, 20899], 
            3:[7248, 20900], 4:[3541, 20901], 5:[3542, 7249, 20902], 6:[7250]}], 
            [{0:3, 1:4, 2:5, 3:6}]]

Previously, for a single list of dictionaries, we solved this by creating an empty dictionary with default values as lists, i.e. we used collections.defaultdict(list).

Given this case is a list of lists, I thought another for loop would be the solution, appending dictionaries into an empty list:

from collections import defaultdict
correct2 = defaultdict(list)

empty = []

for smaller_list in list_of_dictionaries2:
    for d in smaller_list:
        for k,v in d.items():
            correct2[k].append(v)
    empty.append(correct2)

This is very wrong.

>>> print(empty)
[defaultdict(<class 'list'>, {0: [3523, 7245, 3], 1: [3524, 7246, 20898, 4], 
2: [3540, 7247, 20899, 5], 4: [3541, 20901], 5: [3542, 7249, 20902], 
3: [7248, 20900, 6], 6: [7250]}), defaultdict(<class 'list'>, 
{0: [3523, 7245, 3], 1: [3524, 7246, 20898, 4], 2: [3540, 7247, 20899, 5], 
4: [3541, 20901], 5: [3542, 7249, 20902], 3: [7248, 20900, 6], 6: [7250]})]

It looks like all dictionaries were combined. And there are two copies. This is not what I want.

How do I do this for each individual list, like above? Where am I mistaken in my understanding?

score 2 · Accepted Answer · answered Oct 08 '18 at 02:55

2

You don't actually have a list of dictionaries, but a list of lists of dictionaries, and you're trying to merge the dictionaries within the sub-lists, so you should initialize your defaultdict inside the loop that iterates through the main list:

empty = []
for smaller_list in list_of_dictionaries2:
    correct2 = defaultdict(list)
    for d in smaller_list:
        for k,v in d.items():
            correct2[k].append(v)
    empty.append(correct2)

empty would become:

[defaultdict(<class 'list'>, {0: [3523, 7245], 1: [3524, 7246, 20898], 2: [3540, 7247, 20899], 4: [3541, 20901], 5: [3542, 7249, 20902], 3: [7248, 20900], 6: [7250]}), defaultdict(<class 'list'>, {0: [3], 1: [4], 2: [5], 3: [6]})]

Note that your expected output for the second merged defaultdict is incorrect since the value of each key should be a list after merge. Also the list of lists of dictionary should become a list of defaultdicts of lists after the merge, not a list of lists of defaultdicts of lists.

answered Oct 08 '18 at 02:55

blhsing

91,368
6
71
106

I agree with what you are saying but I also believe he wants to see how to get it all into one dict – vash_the_stampede Oct 08 '18 at 03:00
@vash_the_stampede I think blshing's output above looks correct, right? – ShanZhengYang Oct 08 '18 at 03:04
@blshing Thank you for the explanation---I realize now my mistake. A corollary: Why use `correct2= defaultdict(list)` in this case instead of `correct2=dict()`? Just so I understand – ShanZhengYang Oct 08 '18 at 03:05
@ShanZhengYang ha its your question! Not mine :) I thought you were trying to produce one defaultdict that contains everything – vash_the_stampede Oct 08 '18 at 03:05
@vash_the_stampede Sorry; I've tried to edit the question to make it more clear. :) – ShanZhengYang Oct 08 '18 at 03:06

score 1 · Answer 2 · answered Oct 08 '18 at 02:58

you should initialize correct2 every loop like this

for smaller_list in list_of_dictionaries2:
    correct2 = defaultdict(list)
    for d in smaller_list:
        for k, v in d.items():
            correct2[k].append(v)
    empty.append(correct2)

out put will be

[defaultdict(<class 'list'>, {0: [3523, 7245], 1: [3524, 7246, 20898],
2: [3540, 7247, 20899], 4: [3541, 20901], 5: [3542, 7249, 20902], 
3: [7248, 20900], 6: [7250]}), 
defaultdict(<class 'list'>, {0: [3],1: [4], 2: [5], 3: [6]})]

note that defaultdict is mutable Objects

This makes sense---I feel like I'm missing the point maybe. Why use `defaultdict(list)` in this case instead of `correct2=dict()`? Is there a performance difference? — ShanZhengYang, Oct 08 '18 at 03:02

For a list of lists, merge the dictionaries with collections.defaultdict

2 Answers2