Sorting a list of dictionaries by all keys being unique

Question

Pulling my hair out with this one.

I have a list of dictionaries without a unique primary ID key for each unique entry (the dictionary is built on the fly):

dicts = [{'firstname': 'john', 'lastname': 'doe', 'code': 'crumpets'},
         {'firstname': 'john', 'lastname': 'roe', 'code': 'roe'},
         {'firstname': 'john', 'lastname': 'doe', 'code': 'crumpets'},
         {'firstname': 'thom', 'lastname': 'doe', 'code': 'crumpets'},
]

How do I go about filtering out lists of dictionaries like this where any repeating {} within the list are removed? So I need to check if all three of the dictionary keys match up with another in the list...and then discard that from the dict if that check is met.

So, for my example above, the first and third "entries" need to be removed as they are duplicates.

So from the moment there are duplicates, you want to remove *all* elements? — Willem Van Onsem, Jun 19 '17 at 09:59
map the dicts to tuple pairs, create a set of those (which makes them unique) and map them back to dicts — Anthon, Jun 19 '17 at 10:01
No - just remove the duplicates. So if one dictionary in a list is equal to another - drop one of these (but not both). — , Jun 19 '17 at 10:02
possible solution , https://stackoverflow.com/questions/9427163/remove-duplicate-dict-in-list-in-python — Stack, Jun 19 '17 at 10:03

Moses Koledoye · Accepted Answer · 2017-06-19T10:10:19.550

5

You use create frozensets from the dicts and put those in a set to remove dupes:

dcts = [dict(d) for d in set(frozenset(d.items()) for d in dcts)]
print(dcts)

[{'code': 'roe', 'firstname': 'john', 'lastname': 'roe'},
 {'code': 'crumpets', 'firstname': 'thom', 'lastname': 'doe'},
 {'code': 'crumpets', 'firstname': 'john', 'lastname': 'doe'}]

If you choose to remove all entries of the duplicates you can use a counter:

from collections import Counter

dcts = [dict(d) for d, cnt in Counter(frozenset(d.items()) for d in dcts).items() 
                                                                      if cnt==1]
print(dcts)

[{'code': 'roe', 'firstname': 'john', 'lastname': 'roe'},
 {'code': 'crumpets', 'firstname': 'thom', 'lastname': 'doe'}]

edited Jun 19 '17 at 10:10

answered Jun 19 '17 at 10:04

Moses Koledoye

77,341
8
133
139

Will see if I can implement this - but not sure I understand how you can define a dictionary on the fly and then cross check it. Will need to implement this and do some more reading I think. Thank you. – Jun 19 '17 at 10:22
Is there any reason other than onelining this, to not use simple for loop with new container like this: `new_ct = []` and `for item in ct: if item not in new_ct: new_ct.append(item)` ? – Sardorbek Imomaliev Jun 19 '17 at 10:22
@MichaelRoberts First part of his answer does what you want `dcts = [dict(d) for d in set(frozenset(d.items()) for d in dcts)]` – Sardorbek Imomaliev Jun 19 '17 at 10:23
@SardorbekImomaliev Apologies, yes that is correct. I was in the middle of editing my comment when I realised. – Jun 19 '17 at 10:25
To expand upon my above comment, surely dcts is referenced before an assignment? – Jun 19 '17 at 10:28
@MichaelRoberts `dcts` is your list of dicts it is called `dicts` in your question – Sardorbek Imomaliev Jun 19 '17 at 10:28
I see. Yes this works - I should probably have given a different variable name for my dict. Accepted answer - very elegantly done in one line. – Jun 19 '17 at 10:31

Aristide · Answer 2 · 2017-06-19T11:06:44.097

Remove duplicates in a list of non-hashable elements requires you to make them hashable on the fly:

def remove_duplicated_dicts(elements):
    seen = set()
    result = []
    for element in elements:
        element_as_tuple = tuple(element.items())
        if element_as_tuple not in seen:
            seen.add(element_as_tuple)
            result.append(element)
    return result

d = [{'firstname': 'john', 'lastname': 'doe', 'code': "crumpets"},
        {'firstname': 'john', 'lastname': 'roe', 'code': "roe"},
        {'firstname': 'john', 'lastname': 'doe', 'code': "crumpets"},
        {'firstname': 'thom', 'lastname': 'doe', 'code': "crumpets"},
]

print(remove_duplicated_dicts(d))

PS.

Non-obvious differences with the accepted answer of Moses Koledoye (as of 2017-06-19 at 13:00:00):

preservation of the original list order;
faster conversions: dict -> tuple instead of dict -> frozendict -> dict (take it with a grain of salt: I have made no benchmark).

Thank you very much for this answer - although I am very grateful for your answer I have accepted a higher voted answer (I was also working within a function with Django so I didn't want to nest a base dictionary sorting method above Django based methods). But thank you, nonetheless. — , Jun 19 '17 at 10:32

score 1 · Answer 3 · answered Jun 19 '17 at 10:05

Given the values of the dictionary are hashable, we can generate our own uniqness filter:

def uniq(iterable, key = lambda x:x):
    keys = set()
    for item in iterable:
        ky = key(item)
        if ky not in keys:
            yield item
            keys.add(ky)

We can then simply use the filter, like:

list(uniq(dicts,key=lambda x:(x['firstname'],x['lastname'],x['code'])))

The filter maintains the original order, and will - for this example - generate:

>>> list(uniq(dicts,key=lambda x:(x['firstname'],x['lastname'],x['code'])))
[{'code': 'crumpets', 'firstname': 'john', 'lastname': 'doe'},
 {'code': 'roe', 'firstname': 'john', 'lastname': 'roe'},
 {'code': 'crumpets', 'firstname': 'thom', 'lastname': 'doe'}]

Thank you very much for this answer - although I am very grateful for your answer I have accepted a higher voted answer (I was also working within a function with Django so I didn't want to nest a base dictionary sorting method above Django based methods). But thank you, nonetheless. — , Jun 19 '17 at 10:33

Sorting a list of dictionaries by all keys being unique

3 Answers3