Best way to eliminate duplicates in a list, but keep previous relative order

Question

Possible Duplicate:
How do you remove duplicates from a list in Python whilst preserving order?
Does Python have an ordered set?

I find myself doing the following quite frequently:

list_of_items # I have a list of items
set(list_of_items) # I want to remove duplicates

However, in converting to a set, I lose the initial ordering I had in the list, which is important.

What is the best way to remove duplicates in a list AND keep the initial (relative) ordering of the list?

Definitely a duplicate. One of the answers is also a near duplicate of the answer from the duplicate question @Levon mentioned. — istruble, Jun 15 '12 at 23:32

score 7 · Accepted Answer · answered Jun 15 '12 at 23:29

7

This is somewhat of an abuse of list comprehensions, but:

seen = set()
item_list = [seen.add(item) or item for item in item_list if item not in seen]

answered Jun 15 '12 at 23:29

kindall

178,883
35
278
309

What does the `or item` do in the left side of the comprehension? – crypdick Jul 31 '18 at 00:58
`seen.add(item)` returns `None` so `or item` makes the expression evaluate to `item`. – kindall Jul 31 '18 at 03:44

score 0 · Answer 2 · answered Jun 15 '12 at 23:35

Use a dictionary to remove duplicates, use the data as keys and anything as values. If it's in the dict, don't add it again.

d = dict()
for i in range(0, len(list_of_items)):
    if(list_of_items[i] not in d):
        d[list_of_items[i]] = i
        list_of_items2.append(list_of_items[i])

gvl · Answer 3 · 2012-06-15T23:55:50.097

0

Slightly hackish, but ...

item_list = [x[1] for x in sorted(x[::-1] for x in dict(zip(item_list[::-1],range(len(item_list)))).items())][::-1]

edited Jun 15 '12 at 23:55

answered Jun 15 '12 at 23:50

gvl

903
7
16

Best way to eliminate duplicates in a list, but keep previous relative order

3 Answers3