Remove duplicates from a list of dicts

Question

I have a list of dicts like this:

[{'ID': 'a', 'Number': 2}, {'ID': 'b', 'Number': 5} , {'ID': 'a', 'Number': 6}, {'ID': 'a', 'Number': 8}, {'ID': 'c', 'Number': 3}]

I want to remove the dicts that have same key and only keep the one with smallest value. The expected result should be:

[{'ID': 'a', 'Number': 2}, {'Id': 'b', 'Number': 5}, {'ID': 'c', 'Number': 3}]

Nice task. What did you try to solve it? [mre]? – Patrick Artner May 03 '20 at 20:00 — Patrick Artner, May 03 '20 at 20:00
Is the `'ID'`/`'Id'` difference intentional? – Samwise May 03 '20 at 20:02 — Samwise, May 03 '20 at 20:02
Should that be `'ID'` or '`Id'` or are both possible? – Błotosmętek May 03 '20 at 20:02 — Błotosmętek, May 03 '20 at 20:02

Joe Iddon · Accepted Answer · 2020-05-03T20:15:30.590

1

Most efficient solution would be to use a temporary lookup dictionary with keys as IDs and values as the current dict which has the lowest Number corresponding to that ID.

l = [{'ID': 'a', 'Number': 2},
     {'ID': 'b', 'Number': 5}, # note that I corrected a typo Id --> ID
     {'ID': 'a', 'Number': 6},
     {'ID': 'a', 'Number': 8},
     {'ID': 'c', 'Number': 3}]
lookup_dict = {}
for d in l:
    if d['ID'] not in lookup_dict or d['Number'] < lookup_dict[d['ID']]['Number']:
        lookup_dict[d['ID']] = d

output = list(lookup_dict.values())

which gives output as:

[{'ID': 'a', 'Number': 2}, {'ID': 'b', 'Number': 5}, {'ID': 'c', 'Number': 3}]

A piece of advice: given your final data structure, I wonder if you may be better off now representing this final data as a dictionary - with the IDs as keys since these are now unique. This would allow for more convenient data access.

edited May 03 '20 at 20:15

answered May 03 '20 at 20:03

Joe Iddon

20,101
7
33
54

Fair play, should have seen that myself. Need to just convert my lookup_dict at the end... – Joe Iddon May 03 '20 at 20:11
@Błotosmętek Fixed it up now. – Joe Iddon May 03 '20 at 20:14
@Błotosmętek is right, that will give a different result if the value in the first dict is not the smallest. – Chencheng Liu May 03 '20 at 20:14
1

@ChenchengLiu Yeah, I've fixed it now to not add to the `output` list as we go, but just convert the running `lookup_dict` at the end. – Joe Iddon May 03 '20 at 20:16
@ChenchengLiu Switching the `2` and the `8` gives the correct: `[{'ID': 'a', 'Number': 2}, {'ID': 'b', 'Number': 5}, {'ID': 'c', 'Number': 3}]`. – Joe Iddon May 03 '20 at 20:17

Remove duplicates from a list of dicts

1 Answers1