Python Remove duplicates from list of dictionaries based on a value

Question

I have list of dictionaries

vals = [
         {'tmpl_id': 67,  'qty_available': -3.0, 'product_id': 72, 'product_qty': 1.0},     
         {'tmpl_id': 67,  'qty_available': 5.0, 'product_id': 71, 'product_qty': 1.0}
         {'tmpl_id': 69,  'qty_available': 10.0, 'product_id': 74, 'product_qty': 1.0}
       ]

from operator import itemgetter
getvals = operator.itemgetter('tmpl_id')

val.sort(key=getvals)

result = []

for k, g in itertools.groupby(val, getvals):

    result.append(g.next())

val[:] = result

I want to remove duplicate values (tmpl_id) and also based on qty_available is lesser or negative

Output will be like:

vals = [
          {'tmpl_id': 67,  'qty_available': 5.0, 'product_id': 71, 'product_qty': 1.0}
          {'tmpl_id': 69,  'qty_available': 10.0, 'product_id': 74, 'product_qty': 1.0}
       ]

You should show what you have tried and say where you are now stuck. People will help you more if you can show your own efforts... — Serge Ballesta, Dec 08 '15 at 14:45
Read this: http://stackoverflow.com/questions/1165352/calculate-difference-in-keys-contained-in-two-python-dictionaries — Idos, Dec 08 '15 at 14:47

score 5 · Accepted Answer · answered Dec 08 '15 at 14:50

from collections import Counter

vals = [{'tmpl_id': 67,  'qty_available': -3.0, 'product_id': 72, 'product_qty': 1.0},
        {'tmpl_id': 67,  'qty_available': 5.0, 'product_id': 71, 'product_qty': 1.0},
        {'tmpl_id': 69,  'qty_available': 10.0, 'product_id': 74, 'product_qty': 1.0},]

k = [x['tmpl_id'] for x in vals]

new_vals=[]

for i in Counter(k):
    all = [x for x in vals if x['tmpl_id']==i]
    new_vals.append(max(all, key=lambda x: x['qty_available']))

>>> new_vals
[
    {'product_qty': 1.0, 'qty_available': 5.0, 'tmpl_id': 67, 'product_id': 71}, 
    {'product_qty': 1.0, 'qty_available': 10.0, 'tmpl_id': 69, 'product_id': 74}
]

Padraic Cunningham · Answer 2 · 2015-12-08T15:11:35.483

You can store the dicts using the value from "tmpl_id" as the key setting the dict as the value, if you get a dict with a higher 'qty_available' then you replace with the current dict :

def remove_dupes(l, k, k2):
    seen = {} 
    for d in vals:
        v, v2 = d[k], d[k2]
        if v not in seen:
            seen[v] = d
        elif v2 > seen[v][k2]:
            seen[v] = d
    return seen

vals[:] = remove_dupes(vals, "tmpl_id",'qty_available' ).values()

Output:

[{'product_id': 71, 'qty_available': 5.0, 'tmpl_id': 67, 'product_qty': 1.0}, 
{'product_id': 74, 'qty_available': 10.0, 'tmpl_id': 69, 'product_qty': 1.0}]

if you were to use sorted and groupby, you just need sort in reverse and get the first value from each v :

from itertools import groupby
from operator import itemgetter

keys = itemgetter("tmpl_id",'qty_available')

vals[:] = (next(v) for k,v in groupby(sorted(vals, key=keys,reverse=True), 
                 key=itemgetter("tmpl_id")))

print(vals)

reversing the sort will mean the higher 'qty_available' will come first so for unique dicts it will just give you that dict, for repeated tmpl_id's you will get the one with the largest value for qty_available'`.

If you want an inplace sort instead of creating a new list just use vals.sort() and remove the call to sorted

Python Remove duplicates from list of dictionaries based on a value

2 Answers2