23

Fastest way to uniqify a list in Python without preserving order? I saw many complicated solutions on the Internet - could they be faster than simply:

list(set([a,b,c,a]))
nihiser
  • 604
  • 1
  • 15
  • 28
Vojta Rylko
  • 1,442
  • 4
  • 16
  • 29

3 Answers3

26

Going to a set only works for lists such that all their items are hashable -- so e.g. in your example if c = [], the code you give will raise an exception. For non-hashable, but comparable items, sorting the list, then using itertools.groupby to extract the unique items from it, is the best available solution (O(N log N)). If items are neither all hashable, nor all comparable, your only "last ditch" solution is O(N squared).

You can code a function to "uniquify" any list that uses the best available approach by trying each approach in order, with a try/except around the first and second (and a return of the result either at the end of the try clause, or, elegantly, in an else clause of the try statement;-).

Alex Martelli
  • 854,459
  • 170
  • 1,222
  • 1,395
25
set([a, b, c, a])

Leave it in that form if possible.

Matt Joiner
  • 112,946
  • 110
  • 377
  • 526
  • 3
    You can iterate over sets and test for membership in sets, so converting back to list if you don't need order is unnecessary. – Chris Lutz Mar 26 '10 at 23:51
  • 4
    It is worth noting that this assumes that all the elements of the list are **hashable** (see the [Pyhon glossary](http://docs.python.org/glossary.html)) – Rodrigue Jun 08 '11 at 15:40
  • If element order is important in the original list (which I know the question says it is not, but it may be to some readers), this may destroy that order. pylang's answer provides an option that maintains order. – Alex Hall Sep 25 '19 at 05:02
6

This updated post by Peter Bengtsson suggests two of the fastest ways to make a list of unique items in Python 3.6+ are:

# Unordered (hashable items)
list(set(seq))

# Order preserving
list(dict.fromkeys(seq))
pylang
  • 40,867
  • 14
  • 129
  • 121