remove duplicates from value of dicts of list

Question

I have a list like this:

list = [{"name": "name1", "zipcode": "zipcode1", "id": "id1"},{"name": "name2", "zipcode": "zipcode2", "id": "id2"}, {"name": "name1", "zipcode": "zipcode3", "id": "id1"}]

I want to remove dicts which has same ids. I know how to remove duplicates but notice they are not duplicates, they have different zipcodes.

I expect this:

list2 = [{"name": "name1", "zipcode": "zipcode1", "id": "id1"},{"name": "name2", "zipcode": "zipcode2", "id": "id2"}

Does this answer your question? "[List of unique dictionaries](/q/11092511/90527)", "[Removing duplicates in a Python list by id](/q/40690873/90527)", "[How do I remove dicts from a list with duplicate fields in python?](/q/11114358/90527)" — outis, Nov 02 '22 at 21:39

score 0 · Answer 1 · answered Nov 02 '22 at 13:50

Just use set to track seen ids:

seen = set()
alist = [
    {"name": "name1", "zipcode": "zipcode1", "id": "id1"},
    {"name": "name2", "zipcode": "zipcode2", "id": "id2"},
    {"name": "name1", "zipcode": "zipcode3", "id": "id1"}
]
result = []
for obj in alist:
    idx = obj['id']
    if idx not in seen:
        result.append(obj)
    seen.add(idx)
print(result)

Output:

[{'name': 'name1', 'zipcode': 'zipcode1', 'id': 'id1'}, {'name': 'name2', 'zipcode': 'zipcode2', 'id': 'id2'}]

Mehrdad Pedramfar · Accepted Answer · 2022-11-02T13:59:09.543

0

Try this in one line:

list1 = [{"name": "name1", "zipcode": "zipcode1", "id": "id1"},{"name": "name2", "zipcode": "zipcode2", "id": "id2"}, {"name": "name1", "zipcode": "zipcode3", "id": "id1"}]

list({i["name"]: i for i in list1}.values())

This is a little tricky, the point is that dictionaries can not have a same key twice. by making the name as a dictionary key and the rest as values.

By getting the values again. you will achieve what you want.

One more thing:

DO NOT use list as a variable name. list is predefined in python and you will override its own functionality bu doing this. some of common mistakes as a variable name are : list, dict, set, input, ...

edited Nov 02 '22 at 13:59

answered Nov 02 '22 at 13:55

Mehrdad Pedramfar

10,941
4
38
59

Note: this won't work if the order is important – funnydman Nov 02 '22 at 13:57
@funnydman that is completely true. Thanks for adding this useful comment. – Mehrdad Pedramfar Nov 02 '22 at 14:03
very clear and short answer, thank you – saba tananashvili Nov 03 '22 at 09:39

score 0 · Answer 3 · answered Nov 02 '22 at 21:49

from collections import defaultdict
list1 = [{"name": "name1", "zipcode": "zipcode1", "id": "id1"}, {"name": "name2","zipcode": "zipcode2", "id": "id2"}, {"name": "name1", "zipcode": "zipcode3", "id": "id1"}]
temp = defaultdict(list)
for i in range(len(list1)):
    for ele in list1[i]:
        if list1[i][ele] not in temp[ele]:
            temp[str((ele))].append(list1[i][ele])        
print(temp)


> output :{'name': ['name1', 'name2'], 'zipcode': ['zipcode1', 'zipcode2', 'zipcode3'], 'id': ['id1', 'id2']}

in this way your dictionary can be seem more clearly. I used for solution collection defaultdict .

I make a list called temp.
With for loop called the elements of dictionary which are list1[0],list1[1],list1[2]
If values not in temp dictionary, values are appended in temp to make a list of values.
When it is printed, as you can see, a dictionary that looks clearer and better when it works.

I hope my code is useful for you.

You can learn more about defaultdict :

How does collections.defaultdict work?

remove duplicates from value of dicts of list

3 Answers3