Remove duplicates from list of dictionaries within list of dictionaries

Question

I have list:

my_list = [{'date': '10.06.2016', 
            'account': [{'name': 'a'}, 
                        {'name': 'a'}, 
                        {'name': 'b'}, 
                        {'name': 'b'}]}, 
           {'date': '22.06.2016', 
            'account': [{'name': 'a'}, 
                        {'name': 'a'}]}]

I want to remove duplicates from the list of dictionaries in 'account':

my_list = [{'date': '10.06.2016', 
            'account': [{'name': 'a'}, 
                        {'name': 'b'}]}, 
           {'date': '22.06.2016', 
            'account': [{'name': 'a'}]}]

When using set, I get the following error:

TypeError: unhashable type: 'dict'

Can anybody help me with this problem?

score 1 · Answer 1 · answered Jul 26 '16 at 16:03

This structure is probably over complicated, but it gets the job done.

my_list = [{'date': '10.06.2016', 
            'account': [{'name': 'a'}, 
                        {'name': 'a'}, 
                        {'name': 'b'}, 
                        {'name': 'b'}]}, 
           {'date': '22.06.2016', 
            'account': [{'name': 'a'}, 
                        {'name': 'a'}]}]

>>> [{'date': date, 
      'account': [{'name': name} for name in group]
      } for group, date in zip([set(account.get('name') 
                                    for account in item.get('account')) 
                                for item in my_list], 
                               [d.get('date') for d in my_list])]
[{'account': [{'name': 'a'}, {'name': 'b'}], 'date': '10.06.2016'},
 {'account': [{'name': 'a'}], 'date': '22.06.2016'}]

score 0 · Answer 2 · answered Jul 26 '16 at 15:13

Sets can only have hashable members and neither lists nor dicts are - but they can be checked for equality.

you can do

def without_duplicates(inlist):
    outlist=[]
    for e in inlist:
        if e not in outlist:
            outlist.append(e)
    return outlist

this can be slow for really big lists

score 0 · Answer 3 · answered Jul 26 '16 at 16:07

Give this code a try:

for d in my_list:
    for k in d:
        if k == 'account':
            v = []
            for d2 in d[k]:
                if d2 not in v:
                    v.append(d2)
            d[k] = v

This is what you get after running the snippet above:

In [347]: my_list
Out[347]: 
[{'account': [{'name': 'a'}, {'name': 'b'}], 'date': '10.06.2016'},
 {'account': [{'name': 'a'}], 'date': '22.06.2016'}]

score 0 · Accepted Answer · answered Jul 26 '16 at 16:38

def deduplicate_account_names(l):
    for d in l:
        names = set(map(lambda d: d.get('name'), d['account']))
        d['account'] = [{'name': name} for name in names]

# even shorter:
# def deduplicate_account_names(l):
#     for d in l:
#         d['account'] = [{'name': name} for name in set(map(lambda d: d.get('name'), d['account']))]

my_list = [{'date': '10.06.2016',
            'account': [{'name': 'a'},
                        {'name': 'a'},
                        {'name': 'b'},
                        {'name': 'b'}]},
           {'date': '22.06.2016',
            'account': [{'name': 'a'},
                        {'name': 'a'}]}]

deduplicate_account_names(my_list)
print(my_list)

# [ {'date': '10.06.2016',
#    'account': [ {'name': 'a'},
#                 {'name': 'b'} ] },
#   {'date': '22.06.2016',
#    'account': [ {'name': 'a'} ] } ]

Remove duplicates from list of dictionaries within list of dictionaries

4 Answers4

Linked