How to remove duplicate from list of values

Question

I have list of dictionary

dictio =[{'key1':'value1'}, {'key1':'value2'}, {'key1':'value1'}, {'key2':'value4'}, {'key2':'value5'}]
from collections import defaultdict

result = defaultdict(list)
for subd in dictio:
    for k, v in subd.items():
        result[k].append(v)
result

My output

While appending 'value1' appending to 'key1' which is not required

defaultdict(list,
            {'key1': ['value1', 'value2', 'value1'],
             'key2': ['value4', 'value5']})

My Expected

`defaultdict(list,
                {'key1': ['value1', 'value2'],
                 'key2': ['value4', 'value5']})`

This question was already asked [here](https://stackoverflow.com/questions/7961363/removing-duplicates-in-the-lists). — Yovel, Nov 10 '20 at 12:01

score 1 · Answer 1 · answered Nov 10 '20 at 11:54

1

You can turn your list of dict into a set and then in a list again, a set would remove all duplicates.

dictio = list(set([{'key1':'value1'}, {'key1':'value2'}, {'key1':'value1'}, {'key2':'value4'}, {'key2':'value5'}]))

answered Nov 10 '20 at 11:54

JLeno46

1,186
2
14
29

score 1 · Answer 2 · answered Nov 10 '20 at 11:56

You can achieve that very simply, just check if the element is already in:

dictio =[{'key1':'value1'}, {'key1':'value2'}, {'key1':'value1'}, {'key2':'value4'}, {'key2':'value5'}]
from collections import defaultdict

result = defaultdict(list)
for subd in dictio:
    for k, v in subd.items():
        if v not in result[k]:
            result[k].append(v)

score 0 · Answer 3 · answered Nov 10 '20 at 11:51

0

Try this example from https://www.w3schools.com/python/python_howto_remove_duplicates.asp

mylist = ["a", "b", "a", "c", "c"]
mylist = list(dict.fromkeys(mylist))
print(mylist)

answered Nov 10 '20 at 11:51

MustardRecord

305
4
14

score 0 · Accepted Answer · answered Nov 10 '20 at 11:56

Instead of list use set

Ex:

dictio =[{'key1':'value1'}, {'key1':'value2'}, {'key1':'value1'}, {'key2':'value4'}, {'key2':'value5'}]
from collections import defaultdict

result = defaultdict(set)
for subd in dictio:
    for k, v in subd.items():
        result[k].add(v)
result
# --> defaultdict(<class 'set'>, {'key1': {'value2', 'value1'}, 'key2': {'value4', 'value5'}})

How to remove duplicate from list of values

4 Answers4