Removing duplicates without creating new list

Question

I need to remove the duplicates in 2ways, one being able to create a new list which i have done by:

def remove_extras(lst):
    new_list = []
    for number in list:
       if number not in new_list:
           new_list.append(number)
    return new_list

I need a second method where i do not create a new list (order doesn't need to be preserved). I tried using set but it is not the same list as the original one hence doesn't pass the test. anyone has any hints or solutions? thanks

Do you know how to remove an element from a list? That would come in handy here. — Scott Hunter, Oct 11 '17 at 17:01

user2390182 · Accepted Answer · 2017-10-11T10:58:59.213

1

You could reuse your function and use slice assignment:

lst[:] = remove_extras(lst)

More established ways to remove duplicates can be used in the same way:

lst[:] = list(set(lst))  # order not preserved
lst[:] = list(OrderedDict.fromkeys(lst))  # order preserved

lst will be the same list object afterwards.

edited Oct 11 '17 at 10:58

answered Oct 11 '17 at 10:47

user2390182

72,016
6
67
89

Why create a temporary list? – Stefan Pochmann Oct 11 '17 at 10:58
@StefanPochmann I wasn't aware you could use any ol' iterator to assign to the slice. I'll leave it as is since the main points of the mutating nature of the slice assignment and the different methods effect on order preservation are clear anyway. I'll leave it as is... – user2390182 Oct 11 '17 at 11:02

Jonathan DEKHTIAR · Answer 2 · 2017-10-11T10:46:06.453

0

Just do the following:

old_list = [1,1,2,2,3,3]

# create a new list
new_list = list(set(old_list))

# modify old list
old_list = list(set(old_list))

edited Oct 11 '17 at 10:46

answered Oct 11 '17 at 10:45

Jonathan DEKHTIAR

3,456
1
21
42

1

That makes a new list as well – user2390182 Oct 11 '17 at 10:46

khelili miliana · Answer 3 · 2017-10-11T10:53:57.710

0

def remove(lst):
    seen = set()
    def unseen():
        for val in lst:
            if not val in seen:
                yield val
                seen.add(val)
    lst[:] = unseen()
    return lst

input :

l = [1,4,2,5,2,2]

Output :

remove(l)
print l
[1, 4, 2, 5]

edited Oct 11 '17 at 10:53

answered Oct 11 '17 at 10:49

khelili miliana

3,730
2
15
28

@schwobaseggl, Edited – khelili miliana Oct 11 '17 at 10:57

Simon Callan · Answer 4 · 2017-10-11T10:54:56.330

0

If you need the new list to literally be the same list, try

data[:] = list(set(data))

By using the slice operator on the assignment, the list gets updated in place, rather than creating a new list

edited Oct 11 '17 at 10:54

answered Oct 11 '17 at 10:49

Simon Callan

3,020
1
23
34

@StefanPochmann I think I fixed them both. – Simon Callan Oct 11 '17 at 10:56

score 0 · Answer 5 · answered Oct 11 '17 at 10:55

0

lst[:] = set(lst)

Demo:

>>> lst = [1, 2, 3, 1, 2, 3]
>>> lst[:] = set(lst)
>>> lst
[1, 2, 3]

answered Oct 11 '17 at 10:55

Stefan Pochmann

27,593
8
44
107

score 0 · Answer 6 · answered Oct 11 '17 at 11:02

Because you specifically asked to not create a new list object, you can iterate over the set of items in the list, and remove instances from the list where the count of the item is greater than 1.

x_list = [1,1,2,2,3,3]

for x in set(x_list):
    for i in range(1, x_list.count(x)):
        x_list.remove(x)

x_list
[1, 2, 3]

Removing duplicates without creating new list

6 Answers6