How to merge a list of dictionaries with duplicate keys

Question

I have a list of dictionaries:

[
    {
        'Tahsin': [
            {'January': 1}
        ]
    },
    {
        'Arabic Language': [
            {'September': 1}
        ]
    },
    {
        'Arabic Language': [
            {'August': 2}
        ]
    },
    {
        'Arabic Language': [
            {'August': 2}
        ]
    }
]

I want to merge the values under the same keys and remove duplicates.

I tried this following code:

list_of_unique_dicts = []
for dict_ in student_per_course:
    if dict_ not in list_of_unique_dicts:
       list_of_unique_dicts.append(dict_)

and I got the result:

[
    {
        'Tahsin': [
            {'January': 1}
        ]
    }, 
    {'Arabic Language': [
        {'September': 1}
        ]
    },
    {
        'Arabic Language': [
            {'August': 2}
        ]
    }
]

not perfect result, the values of the month still have duplicates.

Then, I tried this following code:

bar = {
        k: [d.get(k) for d in list_of_unique_dicts]
        for k in set().union(*list_of_unique_dicts)
    }

and got this result:

{
    'Tahsin': [
        [
            {'January': 1}
        ],
        None, None
    ], 
    'Arabic Language': [
        None, 
        [
            {'September': 1}
        ],
        [
            {'August': 2}
        ]
    ]
}

Still not the perfect result ^_^.

I also tried using Pandas with following code:

res = pd.DataFrame(list_of_unique_dicts).to_dict(orient='list')

And got this result:

{
    'Tahsin': [
        [
            {'January': 1}
        ], 
        nan, nan
    ], 
    'Arabic Language': [
        nan, 
        [
            {'September': 1}
        ],
        [
            {'August': 2}
        ]
    ]
}

The above still not the result what I want.

The expected result should be:

[
    {
        'Tahsin': [
            {'January': 1}
        ]
    },
    {
        'Arabic Language':
            [
                {'September': 1,
                 'August': 2
                 }
            ]
    },
]

where the values from the first result are merged.

So, how to do that..?, any help would be appreciated :)

One option is to create an intermediate dict (not a list of dicts) similar to your second approach. Loop through all entries in `list_of_unique_dicts`, if the key for this entry doesn't exist in the intermediate dict, add it, otherwise update the value in the intermediate dict. Then at the end, you can convert this intermediate dict into a list of dicts. Try this and post the results. — linus, Sep 21 '19 at 04:15

jezrael · Accepted Answer · 2019-09-21T05:42:14.020

Ide is create set of tuples first with flattening:

L = set([(k, k1, v1) for d in L for k, v in d.items() for y in v for k1, v1 in y.items()])
print (L)
{('Arabic Language', 'August', 2), 
 ('Tahsin', 'January', 1), 
 ('Arabic Language', 'September', 1)}

And then convert back to your structure:

from collections import defaultdict
out = defaultdict(dict)
for a,b,c in L:
    out[a][b] = c

out = [{k: [v] for k, v in out.items()}]
print(out)
[{'Arabic Language': [{'August': 2, 'September': 1}], 'Tahsin': [{'January': 1}]}]

How to merge a list of dictionaries with duplicate keys

1 Answers1