Making an list of dictionary unique is not working in python

Question

I have duplicates in list of dictionary but i could not make it unqiue when i use set in python

>>> b = [
    {"email_address": "aaa", "verify_score": "75"},
    {"email_address": "bbb", "verify_score": "75"},
    {"email_address": "Emailjcb.ab.baseball@gmail.com", "verify_score": "10"},
    {"email_address": "aaa", "verify_score": "75"},
    {"email_address": "carolpaterick@gmail.com", "verify_score": "10"},
    {"email_address": "37a11ce00909300817u2ca1bb5ka40e8422d4bc38b2@mail.gmail.com", "verify_score": "10"},
]
>>> b = set(list(b))
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
TypeError: unhashable type: 'dict'
>>>

Error

Traceback (most recent call last):
      File "<stdin>", line 1, in <module>
    TypeError: unhashable type: 'dict'

How to make this list of dictionary unique

b = [{"email_address": "aaa", "verify_score": "75"}, {"email_address": "bbb", "verify_score": "75"}, {"email_address": "Emailjcb.ab.baseball@gmail.com", "verify_score": "10"}, {"email_address": "aaa", "verify_score": "75"}, {"email_address": "carolpaterick@gmail.com", "verify_score": "10"}, {"email_address": "37a11ce00909300817u2ca1bb5ka40e8422d4bc38b2@mail.gmail.com", "verify_score": "10"}]

Why do you want to do this? (i.e. what is your end goal?) Chances are you are not tackling your problem with the best approach. If you DO need to do this, then I guess you can somehow convert your `dict` to a hashable type like `tuple` first, then use `set`, and finally convert back to `dict`. — Julien, Jul 14 '16 at 05:29
Your dictionaries are all unique because some keys are starting with "verify..." while others start with "verfiy..." — Valera Grishin, Jul 14 '16 at 05:32
Do you want to get rid of an email address that appears twice, or if the email address and score appear twice? — Burhan Khalid, Jul 14 '16 at 05:33
@BurhanKhalid yes exactly this is occuring twice "{"email_address": "aaa", "verify_score": "75"}" — Mounarajan, Jul 14 '16 at 05:35
Possible duplicate of [Python - List of unique dictionaries](http://stackoverflow.com/questions/11092511/python-list-of-unique-dictionaries) — tripleee, Jul 26 '16 at 06:15

maxymoo · Answer 1 · 2016-07-14T05:36:59.797

0

As suggested by Julien in the comments, you can convert to a hashable type like tuple, and then do your unique over that:

>>> set(tuple(d.items()) for d in b)
set([(('verify_score', '10'), ('email_address', 'carolpaterick@gmail.com')), (('verify_score', '10'), ('email_address', '37a11ce00909300817u2ca1bb5ka40e8422d4bc38b2@mail.gmail.com')), (('verify_score', '10'), ('email_address', 'Emailjcb.ab.baseball@gmail.com')), (('verify_score', '75'), ('email_address', 'bbb')), (('verify_score', '75'), ('email_address', 'aaa'))])

edited Jul 14 '16 at 05:36

answered Jul 14 '16 at 05:33

maxymoo

35,286
11
92
119

still results are same – Mounarajan Jul 14 '16 at 05:34
Try: `b = map(dict, set(tuple(d.items()) for d in b))` – taleinat Jul 14 '16 at 05:37
Use frozensets instead of tuples. If you use tuples then the order of the elements in the dict (yes, I really just said order of elements in a dict) can create differences between dicts with the same elements. Try it - run that piece of code a few times and see if it reliably removes all duplicates. – Aran-Fey Jul 14 '16 at 05:48
@Rawing i get the same result each time ... won't the keys be sorted alphabetically? – maxymoo Jul 14 '16 at 05:52
@maxymoo My bad, I read in the documentation that the order is arbitrary, but arbitrary != random. The python3 documentation however does mention this: [`[the order] depends on the dictionary’s history of insertions and deletions`](https://docs.python.org/3/library/stdtypes.html#dictionary-view-objects). I'd use frozensets just to be on the safe side. – Aran-Fey Jul 14 '16 at 06:34

score 0 · Answer 2 · answered Jul 14 '16 at 05:35

Python dictionaries are unhashable which means they are mutable containers. They are not integers or strings that are always the same; the order of contents can change but semantically be the same.

What you could do is try to change the dictionaries into frozensets, or some other hashable type.

>>> unhashable = {'b': 'a', 'a': 'b'}
>>> hashable = frozenset(unhashable.items())

Making an list of dictionary unique is not working in python

2 Answers2