10

I have following list of dictionaries:

d = [
{ 'name': 'test', 'regions': [{'country': 'UK'}] },
{ 'name': 'test', 'regions': [{'country': 'US'}, {'country': 'DE'}] },
{ 'name': 'test 1', 'regions': [{'country': 'UK'}], 'clients': ['1', '2', '5'] },
{ 'name': 'test', 'regions': [{'country': 'UK'}] },
]

What is the easiest way to remove entries from the list that are duplicates ?

I saw solutions that work, but only if an item doesn't have nested dicts or lists

pablox
  • 643
  • 2
  • 8
  • 17

1 Answers1

25

How about this:

new_d = []
for x in d:
    if x not in new_d:
        new_d.append(x)
wim
  • 338,267
  • 99
  • 616
  • 750
  • I knew it is going to be something simple :) Thanks – pablox Jan 23 '12 at 13:35
  • Beware: this works only if the object is hashable. `dict` objects are not hashable and thus you'll get an error if using this approach on objects that contain unhashable objects. – JohnnyUtah Nov 16 '20 at 22:08
  • 1
    @JohnnyUtah That's incorrect. This works with unhashable objects fine. – wim Nov 16 '20 at 22:26
  • @wim Read my statement carefully. I agree this works fine with hashable objects. But if that hashable object *contains* unhashable objects, it will (and does) fail. – JohnnyUtah Dec 20 '20 at 21:26
  • 1
    @JohnnyUtah You are still incorrect. This code works fine even with nested unhashable objects. In fact the example given in the question has nested unhashable objects, and it demonstrably works for that example. – wim Dec 20 '20 at 22:13
  • 1
    It works fine with a list of unhashable objects - dicts in my case, ex: d = [{'a':1,'b':2},{'a':1,'b':2}], set(d) fails and the approach above works ok. – Vitaliy Terziev Oct 31 '21 at 23:44