Remove duplicates from a list with python

Question

I wrote the following python function to remove duplicates from a given list:

def remove_dup(mylist):
    for item in mylist:
        while mylist.count(item) > 1:
            mylist.remove(item)
    return mylist

It seems to work with lists of integer but as soon as I include a string it does not perform as expected. For ex:

ml=[1, 1, 'b', 1, 1, 'a', 45, 2, 2, 'b', 2, 2, 3, 'a', 3, 3, 45]
remove_dup(ml)
['b', 1, 'b', 2, 'a', 3, 45]

So I get 'b' twice in the list. I am new to Python, but why is it?

The main problem in your code is that you remove items from the list while iterating on it - as a general rule, don't do that. You just got 'lucky' if you got the right output with your first test data. — Thierry Lathuille, Feb 07 '21 at 17:12
[How to remove while iterating](https://stackoverflow.com/questions/1207406/how-to-remove-items-from-a-list-while-iterating) — Prune, Feb 07 '21 at 17:23

score 1 · Answer 1 · answered Feb 07 '21 at 17:10

Issue with your code is that you are modifying the list while iterating over it. You can fix your code by iterating over the copy of list.

def remove_dup(mylist):
    for item in list(mylist):
        #        ^ `list(my_list)` will return a new list
        while mylist.count(item) > 1:
            mylist.remove(item)
    return mylist

Sample Run:

>>> ml=[1, 1, 'b', 1, 1, 'a', 45, 2, 2, 'b', 2, 2, 3, 'a', 3, 3, 45]
>>> remove_dup(ml)
[1, 'b', 2, 'a', 3, 45]

You can also refer Removing duplicates in the lists's answers to check better alternatives to achieve this.

Remove duplicates from a list with python

1 Answers1