How to sort and remove duplicates from Python list?

Question

Given a list of strings, I want to sort it alphabetically and remove duplicates. I know I can do this:

from sets import Set
[...]
myHash = Set(myList)

but I don't know how to retrieve the list members from the hash in alphabetical order.

I'm not married to the hash, so any way to accomplish this will work. Also, performance is not an issue, so I'd prefer a solution that is expressed in code clearly to a fast but more opaque one.

Also see [here](http://stackoverflow.com/q/7961363/1129682) for more information — user1129682, Mar 14 '14 at 17:37
This question, after @ColonelPanic's edit, is kind of a mess; the question in the title and the question in the body are not the same. The title indicates that the original order, pre-duplicate-removal, should be preserved. But the body presents a scenario where this is not in fact necessary. — Mark Amery, Sep 29 '19 at 14:05
I changed the title to match the body and the accepted answer. — Vladimir Panteleev, May 03 '22 at 07:55

score 209 · Accepted Answer · edited Sep 26 '12 at 14:22

209

A list can be sorted and deduplicated using built-in functions:

myList = sorted(set(myList))

set is a built-in function for Python >= 2.3
sorted is a built-in function for Python >= 2.4

edited Sep 26 '12 at 14:22

Bengt

14,011
7
48
66

answered Jan 26 '09 at 14:16

Rod Daunoravicius

2,488
1
16
10

17

This does not work if your myList has unhashable objects. – J_Zar Nov 14 '12 at 11:30
wouldn't set(sorted(myList)) be faster? I mean isn't it faster to first sort a list and then remove its duplicates than first removing its duplicates and only sort it afterwards? – Zuzu Corneliu Jan 26 '17 at 19:27
1

@CorneliuZuzu Removing duplicates with set() changes order, so you have to do it this way – Dimali Jun 13 '17 at 09:25
@CorneliuZuzu `set` uses a hash table rather than a tree or sorted list internally, so its speed is unaffected by whether the elements are already sorted. This also means that the items end up in hash order, which isn't the same as sorted order (as Dimali said). – Arthur Tacca May 18 '18 at 09:22
2

Downvoted because there is a distinction between ordered and sorted. Ordered means keep the original order, e.g. f([3,1,4,1,5,9,2,6,5,3,5]) = [3,1,4,5,9,2,6] – Ken Seehart Dec 19 '19 at 03:38
1

@user3667349 The "keep order" clause was not a part of the original question, it was added by the Colonel Panic edit in 2015. – Rod Daunoravicius Dec 24 '19 at 17:18
@ZuzuCorneliu set does not maintain the order so set after sorted will again make the list unordered. – thinkingmonster Nov 18 '21 at 10:15
Ah yes yes, you are correct guys, it slipped my logic at the time ;) – Zuzu Corneliu Nov 28 '21 at 08:28

sykora · Answer 2 · 2009-01-26T15:15:00.710

13

If your input is already sorted, then there may be a simpler way to do it:

from operator import itemgetter
from itertools import groupby
unique_list = list(map(itemgetter(0), groupby(yourList)))

edited Jan 26 '09 at 15:15

answered Jan 26 '09 at 14:48

sykora

96,888
11
64
71

4

This can also be expressed as [e for e, _ in groupby(sortedList)] – Rafał Dowgird Jan 26 '09 at 15:25
This is O(n) rather than O(n log n) right? – Colonel Panic Oct 19 '15 at 14:55
FWIW something similar was added to the list of [recipes](https://docs.python.org/3/library/itertools.html#recipes) from the documentation for `itertools`. – Cristian Ciupitu Feb 16 '16 at 10:58

score 6 · Answer 3 · edited Jul 20 '23 at 14:44

If you want to keep order of the original list, just use OrderedDict with None as values.

In Python2:

from collections import OrderedDict
from itertools import izip, repeat

unique_list = list(OrderedDict(izip(my_list, repeat(None))))

In Python3 it's even simpler:

from collections import OrderedDict
from itertools import repeat

unique_list = list(OrderedDict(zip(my_list, repeat(None))))

If you don't like iterators (zip and repeat) you can use a generator (works both in 2 & 3):

from collections import OrderedDict
unique_list = list(OrderedDict((element, None) for element in my_list))

score 3 · Answer 4 · answered Jan 26 '09 at 14:16

If it's clarity you're after, rather than speed, I think this is very clear:

def sortAndUniq(input):
  output = []
  for x in input:
    if x not in output:
      output.append(x)
  output.sort()
  return output

It's O(n^2) though, with the repeated use of not in for each element of the input list.

score 2 · Answer 5 · answered Jan 26 '09 at 15:22

> but I don't know how to retrieve the list members from the hash in alphabetical order.

Not really your main question, but for future reference Rod's answer using sorted can be used for traversing a dict's keys in sorted order:

for key in sorted(my_dict.keys()):
   print key, my_dict[key]
   ...

and also because tuple's are ordered by the first member of the tuple, you can do the same with items:

for key, val in sorted(my_dict.items()):
    print key, val
    ...

score -1 · Answer 6 · edited Jul 20 '23 at 14:44

-1

For the string data

output = []

    def uniq(input):
        if input not in output:
           output.append(input)
print output

edited Jul 20 '23 at 14:44

Benjamin Loison

3,782
4
16
33

answered Jun 26 '13 at 09:36

user2515605

7
2

How to sort and remove duplicates from Python list?

6 Answers6

Linked

Related