0

I have a ordered list with duplicates (objects that compare equal) and I want to remove the duplicates. As I want to keep the order, I cannot use set.

The duplicates in my list don't ocurr directly after each other and I want to keep the last ocurrence. In this related post, it seems like only the first ocurrence of a duplicate is kept.

As I simplified example, this is what I want:

list_with_duplicates = [1, 2, 1, 3, 2, 1]
list_without_duplicates = [3, 2, 1]

The only thing I can think of is a messy implementation using multiple iterations to first save the last ocurrence of each duplicate and then recreating the original list using the order of the last ocurring duplicates.

Community
  • 1
  • 1
stefanbschneider
  • 5,460
  • 8
  • 50
  • 88

5 Answers5

2

Use any answer you like in the linked duplicate (such as the top one), with one change: reverse your list before you iterate over it, and reverse the result when you're done.

def rem_rev(seq):
    seen = set()
    seen_add = seen.add
    return [x for x in seq[::-1] if not (x in seen or seen_add(x))][::-1]
    #                     ^^^^^^                                   ^^^^^^
    #                  or reversed(seq)
Community
  • 1
  • 1
TigerhawkT3
  • 48,464
  • 6
  • 60
  • 97
  • I don't really get the boolean expression to be honest. What is the `or seen_add(x)` for? I guess it somehow adds `x` to `seen` if it's not already there but I don't know how or what the return value for that would be. – stefanbschneider Dec 19 '16 at 08:41
  • 1
    @CGFoX - For each element in `seq[::-1]`, it only includes that element in the resulting list if it isn't in `seen`. If `x in seen` isn't the case, it checks the result of adding `x` to `seen`, which returns `None`. It's basically "include it if it's not there, or add it if it's not." – TigerhawkT3 Dec 19 '16 at 09:12
2

For Python >= 3.7, dict is ordered, so a simple solution is:

>>> x = [1, 2, 1, 3, 2, 1]
>>> list(reversed(dict.fromkeys(reversed(x))))
[3, 2, 1]

or:

>>> list(dict.fromkeys(x[::-1]))[::-1]
[3, 2, 1]

For earlier Python versions, use OrderedDict instead of dict.

ekhumoro
  • 115,249
  • 20
  • 229
  • 336
1

This could be a clean solution too:

x = [1, 2, 1, 3, 2, 1]
sorted(set(x), key=lambda i: x[::-1].index(i), reverse=True)
# [3, 2, 1]
Mayank Porwal
  • 33,470
  • 8
  • 37
  • 58
0

You can reverse a list using '::-1':

>>> result = []
>>> for item in l[::-1]:
...     if item not in result:
...             result.append(item)
... 
>>> result[::-1]
[3, 2, 1]
>>>
Maurice Meyer
  • 17,279
  • 4
  • 30
  • 47
0
>>> def keep_second(x, y):
>>>     return y
>>> [functools.reduce(keep_second, group) for _, group in itertools.groupby(sorted(items, key=key_func), key=key_func)
yihuang
  • 329
  • 2
  • 7