Find duplicate objects in a dictionary list based on combination of values

Question

I have the following dictionary list

dict_list = [{"feed_id": 101, "query_id": 201, "bind_id": 301, "qname":"q1"},                    
             {"feed_id": 101, "query_id": 201, "bind_id": 301, "qname":"q2"},                    
             {"feed_id": 103, "query_id": 201, "bind_id": 301, "qname":"q1"},                    
             {"feed_id": 103, "query_id": 202, "bind_id": 301, "qname":"q3"},                    
             {"feed_id": 103, "bind_id": 301, "bname": "b1"},                                     
             {"feed_id": 101, "query_id": 201, "qname":"q1"}]

I want to remove duplicates based on the combination of three keys. The result list should look like

result_dict_list = [{"feed_id": 101, "query_id": 201, "bind_id": 301, "qname":"q1"},            
                    {"feed_id": 103, "query_id": 201, "bind_id": 301, "qname":"q1"},            
                    {"feed_id": 103, "query_id": 202, "bind_id": 301, "qname":"q3"},            
                    {"feed_id": 103, "bind_id": 301, "bname": "b1"},                             
                    {"feed_id": 101, "query_id": 201, "qname":"q1"}]

These are the requirements in terms of object structure

feed_id will exist for all objects and cannot be null
query_id and bind_id are optional. If even one of the property does not exist, no need to check whether the object is duplicate
All three properties values are numbers. If the key exist, value cannot be null.
There can be many other properties in each object but to eliminate duplicates we care only about feed_id, query_id and bind_id
Order of the list doesn't matter

What would be the most efficient way to remove duplicates from the list in python?

Thanks

Does this answer your question? [Remove duplicate dict in list in Python](https://stackoverflow.com/questions/9427163/remove-duplicate-dict-in-list-in-python) — Missilexent, May 12 '20 at 22:03
I missed a couple of points in the requirements. Made an update. — Gowthaman, May 13 '20 at 02:56

score 2 · Answer 1 · answered May 12 '20 at 21:46

It's fairly straightforward:

dict_list = [{"feed_id": 101, "query_id": 201, "bind_id": 301},                    
             {"feed_id": 101, "query_id": 201, "bind_id": 301},                    
             {"feed_id": 103, "query_id": 201, "bind_id": 301},                    
             {"feed_id": 103, "query_id": 202, "bind_id": 301},                    
             {"feed_id": 103, "bind_id": 301},                                     
             {"feed_id": 101, "query_id": 201}]
result_dict_list = []
for d in dict_list:
    if d not in result_dict_list:
        result_dict_list.append(d)

print(result_dict_list)

I missed a requirement and I have updated the question. Thanks for the reply! — Gowthaman, May 13 '20 at 03:17

Gowthaman · Accepted Answer · 2020-05-13T17:51:10.977

I found a working solution for now. But wanted to check if there is a better way to do it. This is what I have.

list_with_all_keys = []
properties = ['feed_id', 'query_id', 'bind_id']

for obj in dict_list:
    if all(key in obj for key in properties):
        list_with_all_keys.append(obj)

dict_with_no_duplicates = {tuple(d[k] for k in properties): d 
                                      for d in list_with_all_keys}

list_with_missing_keys = [i for i in dict_list if i not in list_with_all_keys]

result_dict = dict_with_no_duplicates.values() + list_with_missing_keys

Pyfiddle for reference https://pyfiddle.io/fiddle/268c14a1-ccd9-471c-8c61-18ae90d81f47/?i=true

Find duplicate objects in a dictionary list based on combination of values

2 Answers2