How to remove duplicate dictionary Objects from list

Question

I want to remove the dictionary from the list which contains duplicate values

list = [{'data': 'sql_data'},{'data':'mongo_data'},{'data':'sql_data'}]

list = [{'data': 'sql_data'},{'data':'mongo_data'}]

You want to remove duplicate values, duplicate keys or duplicate dicts? — Eric Duminil, Feb 07 '19 at 07:39

U13-Forward · Answer 1 · 2019-02-07T08:05:45.490

2

Use sorted with set:

import ast
print(sorted(map(ast.literal_eval,set(map(str,list))),key=list.index))

Or:

l=[]
for i in list:
    if i not in l:
        l.append(i)

Both Output:

[{'data': 'sql_data'},{'data':'mongo_data'}]

edited Feb 07 '19 at 08:05

answered Feb 07 '19 at 07:31

U13-Forward

69,221
14
89
114

Ah, Wouldn't this now produce: `TypeError: 'type' object is not iterable` – DirtyBit Feb 07 '19 at 07:48
@user5173426 Why? – U13-Forward Feb 07 '19 at 07:49
1

@user5173426 forgot to rename list https://pyfiddle.io/fiddle/9e836cca-81d9-4831-9674-647a84b40a63/?i=true – U13-Forward Feb 07 '19 at 07:52

score 2 · Accepted Answer · answered Feb 07 '19 at 07:32

2

Use pandas, if it doesn't have nested dict

import pandas as pd
list = pd.DataFrame(list).drop_duplicates().to_dict(orient='records')

answered Feb 07 '19 at 07:32

Bimal Gangawal

385
2
17

DirtyBit · Answer 3 · 2019-02-07T07:41:01.647

2

s = [{'data': 'sql_data'},{'data':'mongo_data'},{'data':'sql_data'}]

new = []
for x in s:
    if x not in new: new.append(x)        
print(new)

OR

one-line:

print([i for n, i in enumerate(s) if i not in s[n + 1:]])

OUTPUT:

[{'data': 'sql_data'}, {'data': 'mongo_data'}]

edited Feb 07 '19 at 07:41

answered Feb 07 '19 at 07:33

DirtyBit

16,613
4
34
55

score 2 · Answer 4 · answered Feb 07 '19 at 07:36

Try this, it works even when the key is not 'data':

import json

list = [{'data': 'sql_data'}, {'data': 'mongo_data'}, {'data': 'sql_data'}, {'user': 'admin'}, {'user': 'admin'}]
list_deduplicated = [json.loads(i) for i in set([json.dumps(j) for j in list])]
print(list_deduplicated)

The output will be:

[{'data': 'mongo_data'}, {'data': 'sql_data'}, {'user': 'admin'}]

score 1 · Answer 5 · answered Feb 07 '19 at 07:37

Use list comprehensions to solve this -

my_list=[{'data': 'sql_data'},{'data':'mongo_data'},{'data':'sql_data'}]
unique_list = [e for i, e in enumerate(my_list) if my_list.index(e) == i ]
print(unique_list)
    [{'data': 'sql_data'}, {'data': 'mongo_data'}]

How to remove duplicate dictionary Objects from list

5 Answers5