Remove duplicates from a list

Question

I wrote the following code to remove duplicates from a list:

def remove_duplicates(listy):
    new_listy = []
    for i in range(len(listy)):
        a = listy.pop(i)
        while a in listy:
            listy = listy.remove(a)

        else: new_listy.append(a)
    return new_listy

The following error is given when running the code:

Traceback (most recent call last):
  File "<pyshell#93>", line 1, in <module>
    remove_duplicates([1,1,2,2])
  File "C:\Python27\MIT_4.py", line 4, in remove_duplicates
    a = listy.pop(i)
AttributeError: 'NoneType' object has no attribute 'pop.

What is causing this error? How can it be fixed to solve this correctly?

Pay attention, that when an item removed from listy, len(listy) is changed; usually we loop in reverse: from len(listy) - 1 downto 0 — Dmitry Bychenko, Aug 07 '14 at 06:11
Do you care about the order of the elements? If not, you can make a `set` out of the original list, then make a list out of that. — juanchopanza, Aug 07 '14 at 06:12

score 2 · Answer 1 · answered Aug 07 '14 at 06:36

To remove duplicates, you can use the set function if ordering is not important. But if your intention is to learn python, then it is important to know what is wrong with your code.

This is very unpythonic: for i in range(len(listy)):

Use instead, for item in listy:

Using a = listy.pop(i) this way will cause problems because within the loop list elements are being removed. Elements which were there at the start of the loop will no longer be available when pop() executes..

Remove function does not return a list! So this is wrong: listy = listy.remove(a)

There is no need to remove from listy since you are making a new list anyway. If you wish to update listy, assign the function's return value to listy.

One way to remove duplicates and return a new list preserving the original order:

def remove_duplicates(listy):
    new_listy = []
    for item in listy:
        if item not in new_listy:
            new_listy.append(item)
    return new_listy

I did find one here: http://stackoverflow.com/questions/10549345/how-to-remove-duplicate-items-from-a-list-using-list-comprehension — coder.in.me, Aug 07 '14 at 07:00

score 0 · Answer 2 · answered Aug 07 '14 at 06:11

0

to remove duplicates you can use set function

lst = [1,1,1,2,3,4]
print list(set(lst))

output

[1, 2, 3, 4]

answered Aug 07 '14 at 06:11

Selva

976
1
10
23

1

Note that the retained order here is a "fluke"; if order is important, this will not work in the general case. – jonrsharpe Aug 07 '14 at 06:51

score 0 · Answer 3 · answered Aug 07 '14 at 06:23

Try this. I have rearranged your function a bit and it will give you the desired result

list1 = [1,1,2,2,3]
def remove_duplicates(listy):
    new_listy = []
    for i in listy:
        if i not in new_listy:
            new_listy.append(i)
    return new_listy
print remove_duplicates(list1)

Output: [1,2,3]

score 0 · Answer 4 · answered Aug 07 '14 at 06:24

lista.remove does not create a new list, so you cannot assign the result like this:

listy = listy.remove(a)

Then listy is None and you get errors trying to call pop() on None instead of the list.

Also there are other errors in your code. I suggest reading the documentation for these commands so you understand them better.

Remove duplicates from a list

4 Answers4