13

Possible Duplicate:
Python: removing duplicates from a list of lists

Say i have list 

a=[1,2,1,2,1,3]

If all elements in a are hashable (like in that case), this would do the job: 

list(set(a))

But, what if 

a=[[1,2],[1,2],[1,3]]

?

Community
  • 1
  • 1
Sanich
  • 1,739
  • 6
  • 25
  • 43
  • 1
    These you could turn into tuples before adding to the set. – MattH May 28 '12 at 12:09
  • 1
    possible duplicate of [Python: removing duplicates from a list of lists](http://stackoverflow.com/questions/2213923/python-removing-duplicates-from-a-list-of-lists) http://stackoverflow.com/questions/480214/how-do-you-remove-duplicates-from-a-list-in-python-whilst-preserving-order http://stackoverflow.com/questions/479897/how-do-you-remove-duplicates-from-a-list-in-python – jamylak May 28 '12 at 12:14
  • Just realized this is a duplicate of many other questions, it should be closed. – jamylak May 28 '12 at 12:14
  • The example in the question is a duplicate but the question is not duplicate. Consider other unhashable types such as a list containing duplicate pandas dataframes. – Eric O. Sep 01 '22 at 15:45

2 Answers2

15

Python 2

>>> from itertools import groupby
>>> a = [[1,2],[1,2],[1,3]]
>>> [k for k,v in groupby(sorted(a))]
[[1, 2], [1, 3]]

Works also in Python 3 but with caveat that all elements must be orderable types.

jamylak
  • 128,818
  • 30
  • 231
  • 230
  • This won't work if the elements cannot be compared (e.g. `[[1,2], [1,2], ['bug']]`). But maybe you could do `sorted(a, key=repr)` to fix that. – ekhumoro Mar 03 '19 at 00:55
  • @ekhumoro Ah i get what you mean. You are talking about how in Python 3 `sorted` will give `unorderable types` in that case. Back when i answered this question Python 3 was not popular so this was a Python 2 answer. I'll update to mention that – jamylak Mar 04 '19 at 03:24
0

This set comprehension works for the List of Lists to produce a set of tuples:

>>> {(tuple(e)) for e in a}
set([(1, 2), (1, 3)])

Then use that to turn it into a list of lists again with no duplicates:

>>> [list(x) for x in {(tuple(e)) for e in a}]
[[1, 2], [1, 3]]
the wolf
  • 34,510
  • 13
  • 53
  • 71