30

I have a list of dictionaries

l = [
{'firstname': 'joe', 'surname': 'bloggs'},
{'firstname': 'john', 'surname': 'smith'},
{'firstname': 'joe', 'surname': 'bloggs'},
{'firstname': 'jane', 'surname': 'bloggs'}
]

how do i remove duplicates i.e. {'firstname': 'joe', 'surname': 'bloggs'} appears twice so would want it only appearing once?

John
  • 21,047
  • 43
  • 114
  • 155
  • 4
    What have you already tried? We're not here to just give you code. – Delan Azabani Aug 17 '11 at 09:45
  • Actually, I've looked in Google for some very simple problems that I know can be done easily in Python. And StackOverflow has most definitely provided "just the code" in my assistance, which is all I really need. – Bobort Nov 15 '15 at 23:06

2 Answers2

66

Something like this should do the stuff : 

result = [dict(tupleized) for tupleized in set(tuple(item.items()) for item in l)]

first, I transform the inital dict in a list of tuples, then I put them into a set (that removes duplicates entries), and then back into a dict.

Cédric Julien
  • 78,516
  • 15
  • 127
  • 132
14
import itertools

import operator

import pprint

l = [

{'firstname': 'joe', 'surname': 'bloggs'},

{'firstname': 'john', 'surname': 'smith'},

{'firstname': 'joe', 'surname': 'bloggs'},

{'firstname': 'jane', 'surname': 'bloggs'}

]

getvals = operator.itemgetter('firstname', 'surname')

l.sort(key=getvals)

result = []

for k, g in itertools.groupby(l, getvals):
    result.append(next(g))

l[:] = result

pprint.pprint(l)
Fabio Ramalho
  • 161
  • 1
  • 9
Turkesh Patel
  • 161
  • 6
  • this is a nice solution if you have specific fields you want to de-duplicate on, and don't want to de-dup on all fields present in the dictionary. – Banjer Oct 16 '14 at 13:52