Removing dicts with duplicate value from list of dicts

Question

I have a list of dicts as follows:

[{'ppm_error': -5.441115144810845e-07, 'key': 'Y7', 'obs_ion': 1054.5045550349998},
{'ppm_error': 2.3119997582222951e-07, 'key': 'Y9', 'obs_ion': 1047.547178035},
{'ppm_error': 2.3119997582222951e-07, 'key': 'Y9', 'obs_ion': 1381.24928035},
{'ppm_error': -2.5532659838679713e-06, 'key': 'Y4', 'obs_ion': 741.339467035},
{'ppm_error': 1.3036219678359603e-05, 'key': 'Y10', 'obs_ion': 1349.712302035},
{'ppm_error': 3.4259216556970878e-06, 'key': 'Y6', 'obs_ion': 941.424286035},
{'ppm_error': 1.1292770047090912e-06, 'key': 'Y2', 'obs_ion': 261.156025035},
{'ppm_error': 1.1292770047090912e-06, 'key': 'Y2', 'obs_ion': 389.156424565},
{'ppm_error': 9.326980606898406e-06, 'key': 'Y5', 'obs_ion': 667.3107950350001}
]

I want to remove dicts with duplicate keys such that only dicts with unique 'key' remain. It doesn't matter which dict ends up in the final list. So the final list should look as follows:

[{'ppm_error': -5.441115144810845e-07, 'key': 'Y7', 'obs_ion': 1054.5045550349998},
{'ppm_error': 2.3119997582222951e-07, 'key': 'Y9', 'obs_ion': 1381.24928035},
{'ppm_error': -2.5532659838679713e-06, 'key': 'Y4', 'obs_ion': 741.339467035},
{'ppm_error': 1.3036219678359603e-05, 'key': 'Y10', 'obs_ion': 1349.712302035},
{'ppm_error': 3.4259216556970878e-06, 'key': 'Y6', 'obs_ion': 941.424286035},
{'ppm_error': 1.1292770047090912e-06, 'key': 'Y2', 'obs_ion': 261.156025035},
{'ppm_error': 9.326980606898406e-06, 'key': 'Y5', 'obs_ion': 667.3107950350001}
]

Is it possible to use itertools.groupby function for doing this or is there another way of approaching this problem? Any suggestions?

Also, does the order of dictionaries matter in the output list? — thefourtheye, Apr 05 '14 at 02:54

score 6 · Accepted Answer · answered Apr 05 '14 at 02:56

6

If the order matters, then you can use collections.OrderedDict to collect all the items, like this

from collections import OrderedDict
print OrderedDict((d["key"], d) for d in my_list).values()

And if the order doesn't matter, you can use a normal dictionary, like this

print {d["key"]:d for d in my_list}.values()

answered Apr 05 '14 at 02:56

thefourtheye

233,700
52
457
497

Best solution, as key has that magic property of having unique keys :) – Tadeck Apr 05 '14 at 09:06

score 2 · Answer 2 · answered Apr 05 '14 at 09:04

Another solution is to remember processed keys and return different result if key has already been seen. This can be done using memoization:

def get_key_watcher():
    keys_seen = set()
    def key_not_seen(d):
        key = d['key']
        if key in keys_seen:
            return False  # key is not new
        else:
            keys_seen.add(key)
            return True  # key seen for the first time
    return key_not_seen

Then you could use it like this:

>>> filtered_dicts = filter(get_key_watcher(), dicts)
>>> filtered_dicts
[{'ppm_error': -5.441115144810845e-07, 'obs_ion': 1054.5045550349998, 'key': 'Y7'},
 {'ppm_error': 2.3119997582222951e-07, 'obs_ion': 1047.547178035, 'key': 'Y9'},
 {'ppm_error': -2.5532659838679713e-06, 'obs_ion': 741.339467035, 'key': 'Y4'},
 {'ppm_error': 1.3036219678359603e-05, 'obs_ion': 1349.712302035, 'key': 'Y10'},
 {'ppm_error': 3.4259216556970878e-06, 'obs_ion': 941.424286035, 'key': 'Y6'},
 {'ppm_error': 1.1292770047090912e-06, 'obs_ion': 261.156025035, 'key': 'Y2'},
 {'ppm_error': 9.326980606898406e-06, 'obs_ion': 667.3107950350001, 'key': 'Y5'}]

It maintains order of dictionaries, obviously. And keeps the dictionary encountered first.

Really elegant, I was groping around with search terms like 'filter', and here's a great example of exactly what I was looking for. Just another reminder that python looks an awful lot like pseudocode, which I guess is the definition of intuitive. — Sacrilicious, Mar 18 '15 at 03:00

Stephan Kulla · Answer 3 · 2014-04-05T03:40:46.650

0

I would do it like this:

list = [...] # your list

finallist = dict(map(lambda x: (x['key'],x), list)).values()

Basically its the same solution @thefourtheye provides in his answer...

edited Apr 05 '14 at 03:40

answered Apr 05 '14 at 02:57

Stephan Kulla

4,739
3
26
35

Could you add a description? Currently this answer is incomplete. – bjb568 Apr 05 '14 at 03:35

Joran Beasley · Answer 4 · 2014-04-05T03:09:10.240

0

convert it to a numpy array

a = numpy.array([(d["ppm_error"],d["key"],d["obs_ion"]) for d in my_dicts])
mask =numpy.unique(a[:,1],True)[1]
uniques = a[mask]

then back to a dict

unique_entries = map(dict,[zip(labels,row) for row in uniques])

edited Apr 05 '14 at 03:09

answered Apr 05 '14 at 03:01

Joran Beasley

110,522
12
160
179

Removing dicts with duplicate value from list of dicts

4 Answers4

Linked