Delete duplicates from a list with many Elements

Question

I have a lists with several elements and i'd like to remove the duplicated, here's an example of a list:

list = [{'status': 'OK', 'mid': '6639'}, {'status': 'OK', 'mid': '6639'}, {'status': 'OK', 'mid': '6640'}, {'status': 'OK', 'mid': '6640'}, {'status': 'OK', 'mid': '6641'}, {'status': 'OK', 'mid': '6641'}, {'status': 'OK', 'mid': '6642'}, {'status': 'OK', 'mid': '6643'}]

I have tried to use the function set but i have an error it says that the list is unhashable, does someone have an idea of how to remove the duplications ?

You could make those dictionaries hashable: http://stackoverflow.com/questions/1151658/python-hashable-dicts — poke, Apr 23 '13 at 15:31
See this question: http://stackoverflow.com/questions/9427163/remove-duplicate-dict-in-list-in-python — Nicolas, Apr 23 '13 at 15:31

score 4 · Accepted Answer · answered Apr 23 '13 at 15:31

You can still use a set() but you need to turn each list into a tuple; here we use a sorted tuple of the keys and values to find duplicates accurately:

set(tuple(sorted(el.items())) for el in somelist)

To keep the list in order and keep your dictionaries intact, you could use:

seen = set()
seen_add = seen.add
[x for x in somelist if tuple(sorted(x.items())) not in seen and not seen_add(tuple(sorted(x.items())))]

Demo for the latter:

>>> seen = set()
>>> seen_add = seen.add
>>> [x for x in somelist if tuple(sorted(x.items())) not in seen and not seen_add(tuple(sorted(x.items())))]
[{'status': 'OK', 'mid': '6639'}, {'status': 'OK', 'mid': '6640'}, {'status': 'OK', 'mid': '6641'}, {'status': 'OK', 'mid': '6642'}, {'status': 'OK', 'mid': '6643'}]

is there another solution faster/easier without using the set function ?? — Kirk ERW, Apr 23 '13 at 15:35
@SkivlageLatif: You *have* to detect duplicates somehow, and a set is the most efficient method. — Martijn Pieters, Apr 23 '13 at 15:37

Delete duplicates from a list with many Elements

1 Answers1