How to remove the duplicate dictionaries from the list of dictionaries where dictionary contains an another dictionary?

Question

I have a bit complex list of dictionaries which looks like

[
    {'Name': 'Something XYZ', 'Address': 'Random Address', 'Customer Number': '-', 'User Info': [{'Registration Number': '17002', 'First Name': 'John', 'Middle Name': '', 'Last Name': 'Denver'}, {'Registration Number': '27417', 'First Name': 'Robert', 'Middle Name': '', 'Last Name': 'Patson'}]}, 
    {'Name': 'Something XYZ', 'Address': 'Random Address', 'Customer Number': '-', 'User Info': [{'Registration Number': '27417', 'First Name': 'Robert', 'Middle Name': '', 'Last Name': 'Patson'}, {'Registration Number': '17002', 'First Name': 'John', 'Middle Name': '', 'Last Name': 'Denver'}]}
]

Expected is below

[
    {'Name': 'Something XYZ', 'Address': 'Random Address', 'Customer Number': '-', 'User Info': [{'Registration Number': '17002', 'First Name': 'John', 'Middle Name': '', 'Last Name': 'Denver'}, {'Registration Number': '27417', 'First Name': 'Robert', 'Middle Name': '', 'Last Name': 'Patson'}]}, 
]

I want to remove the duplicate dictionaries in this list but I don't know how to deal with User Info because the order of the items might be different. A duplicate case would be where all the dictionary items are exactly the same and in the case of User Info order doesn't matter.

My first guess would be to transform the list to a set. Just not sure if it works for a list of dicts — Manu, Jun 30 '22 at 08:34
@Manu Yeah! The issue could be with the order of items in the list of dicts. — Nik, Jun 30 '22 at 08:35
What is considered duplicate? `User Info` is the same, but `Name` is different. What is the expected result? — Guy, Jun 30 '22 at 08:35
@Guy Duplicates are when all the dictionary items are the same, in the case of `User Info` order doesn't matter. — Nik, Jun 30 '22 at 08:38
Order doesnt really matter when trying to eliminate duplicates from an unordered collection like a dict. In this case, if my approach actually works, no element would be eliminated because theyre not 100% equal. (EDIT: seems like since Python 3.7, dicts are indeed ordered...) — Manu, Jun 30 '22 at 08:40
@Tomerikoo the idea is that the lists used as values for the `'User Info'` key contain the same dicts, but in a different order. — Karl Knechtel, Jun 30 '22 at 08:43

Mehdi · Accepted Answer · 2022-06-30T10:20:56.720

1

I think the best way is to make a hash of User Info by sum the hash values of it's elements (sum will tolerate position change).

def deepHash(value):
    if type(value) == list:
        return sum([deepHash(x) for x in value])
    
    if type(value) == dict:
        return sum([deepHash(x) * deepHash(y) for x, y in value.items()])

    return hash(str(value))

and you can simply check the hash of you inputs:

assert deepHash({"a": [1,2,3], "c": "d"}) == deepHash({"c": "d", "a": [3,2,1]})

edited Jun 30 '22 at 10:20

answered Jun 30 '22 at 08:42

Mehdi

683
5
16

1

I can't see how that answers the question. Can you please show how it solves the question asked here. Right now this is just a complex way of doing `set(['a', 'b', 'c']) == set(['b', 'c', 'a'])`... – Tomerikoo Jun 30 '22 at 08:53
Just to clarify, what I meant is that you simply show here how to check if two lists of unordered strings (which are hashable and immutable) are the same or not. The question asked here is far from being that simple - it's about checking if two dicts with values of a list of dicts are the same – Tomerikoo Jun 30 '22 at 08:57
@Tomerikoo you have to make a recursive method to compute sum of hash of values and keys and check them all together. i assumed you asked for an idea instead of code. if you want the code i can write it for you in a few minutes. – Mehdi Jun 30 '22 at 10:10

How to remove the duplicate dictionaries from the list of dictionaries where dictionary contains an another dictionary?

1 Answers1