Why is list.remove only removing every second item?

Question

In my Python 2.7.2 IDLE interpreter:

>>> mylist = [1, 2, 3, 4, 5]
>>> for item in mylist:
        mylist.remove(item)

>>> mylist
[2, 4]

Why?

btw, I did see on another question that the proper way to do this is del mylist[:] — Scruffy, May 09 '13 at 15:05

mgilson · Accepted Answer · 2013-05-09T15:16:55.073

It's because when you iterate over a list, python keeps track of the index in the list. Consider the following code instead:

for i in range(len(mylist)):
    if i >= len(mylist):
       break
    item = mylist[i]
    mylist.remove(item)

If we track this (which is essentially what python is doing in your code), then we see that when we remove an item in the list, the number to the right shifts one position to the left to fill the void left when we removed the item. The right item is now at index i and so it will never actually get seen in the iteration because the next thing that happens is we increment i for the next iteration of the for loop.

Now for something a little clever. If instead we iterate over the list backward, we'll clear out the list:

for item in reversed(mylist):
    mylist.remove(item)

The reason here is that we're taking an item off the end of the list at each iteration of the for loop. Since we're always taking items off the end, nothing needs to shift (assuming uniqueness in the list -- If the list isn't unique, the result is the same, but the argument gets a bit more complicated).

Of course, If you're looking to remove all the items from a list, you can do that really easily:

del mylist[:]

or even with slice assignment:

mylist[:] = []

(I mention the latter because it can be useful to replace segments of a list with other items which don't even need to be the same length).

I love Python but there are some strange gotchyas that require a fairly deep understanding of the underlying implementation/concepts. — Scruffy, May 09 '13 at 15:08

score 3 · Answer 2 · answered May 09 '13 at 15:04

3

That's because you're modifying the list while iterating over it, iterate over a shallow copy instead:

>>> mylist = [1, 2, 3, 4, 5]
>>> for item in mylist[:]:      #use mylist[:] or list(mylist)
            mylist.remove(item)
...     
>>> mylist
[]

answered May 09 '13 at 15:04

Ashwini Chaudhary

244,495
58
464
504

score 2 · Answer 3 · answered May 09 '13 at 15:04

2

You are modifying your list while you are looping through it, which is very bad practice.

answered May 09 '13 at 15:04

Pep_8_Guardiola

5,002
1
24
35

Stefano Sanfilippo · Answer 4 · 2013-05-09T21:28:59.983

2

Problem is that you are altering the list while iterating on it. Use a list comprehension instead:

mylist = [1, 2, 3, 4, 5]
mylist = [x for x in mylist if condition(x)]

edited May 09 '13 at 21:28

answered May 09 '13 at 15:05

Stefano Sanfilippo

32,265
7
79
80

1

It is not undefined behavior. – mgilson May 09 '13 at 15:06
The Python specification does not mention what behaviour will be adopted in this corner case, so it's an implementation detail, thus undefined. If I want, I can write a Python VM that explodes upon this condition and it would still be conformant. – Stefano Sanfilippo May 09 '13 at 15:11
1

The specification *does* specify exactly how this works. I don't remember where, but at one point I remember searching for it and finding it explicitly spelled out. – mgilson May 09 '13 at 15:12
See the Note. http://docs.python.org/2/reference/compound_stmts.html#the-for-statement – mgilson May 09 '13 at 15:15
The note (which is that, a note) probably refers to cPython implementation, as the specification does not detail how a list should be explored in terms of internals: *"Each item in turn is assigned to the target list"*, no reference to the internal counter (some lines above). In Iron Python, this could translate to a C# `foreach`, which could have different behaviour. – Stefano Sanfilippo May 09 '13 at 15:20
2

I disagree. Everything that is Cpython implementation dependent in the specification states that explicitly saying "Cpython implementation detail ...". This is a constraint on the implementation of `MutableSequence.__iter__` – mgilson May 09 '13 at 15:23
The only reference I could find (although I admit I did not do a deep search), is the one to [`__iter__`](http://docs.python.org/release/3.2/reference/datamodel.html#object.__iter__) and [`Iterator`](http://docs.python.org/release/3.2/library/stdtypes.html#typeiter), where I do not see any specific reference to the internal mechanism of iteration over mutable types. – Stefano Sanfilippo May 09 '13 at 16:01
I agree with mgilson here. I see this explicitly mentioned in the [`for` statement documentation](http://docs.python.org/2/reference/compound_stmts.html#grammar-token-for_stmt). Note that the specification states an iterator is created for the result of the `expression_list`. I fail to see how any implementation can avoid this behaviour; e.g. adjust the iterator to compensate for deletions on the underlying mutable sequence in a loop body (or elsewhere). – Martijn Pieters May 09 '13 at 17:18
I dropped a line to Guido Van Rossum, his answer will be the last word. I will let you know. – Stefano Sanfilippo May 09 '13 at 17:33
I think a simple email to the Python dev list would have sufficed, there are more people that could have given you a canonical answer to this question. Moreover, the Python documentation always makes it explicit when something is a CPython implementation detail. The interaction between `for` and a list iterator is *not* an implementation detail. The documentation does not state it is an implementation detail, that note is there to point out this interaction exists because new users fall into the trap often enough. – Martijn Pieters May 09 '13 at 17:52
Fact is that during this discussion I asked on #python at Freenode, and two developers said that actually it was undefined, so I wanted to get an official statement. I'm no megalomaniac, he is kind when answering and faster than the ML :) Guido said that it is well defined, indeed. So I'm correcting the answer. Thank you for making me reflect on this. [Here you](http://paste.debian.net/3255/) can find the redacted email he wrote me, if you care. – Stefano Sanfilippo May 09 '13 at 21:28

Kousik · Answer 5 · 2013-05-10T12:39:43.683

>>> mylist = [1, 2, 3, 4, 5]

>>> for item in mylist:
        mylist.remove(item)

in this loop : 1st time it will take 0th element (1) as an item and remove it from mylist.so after that mylist will be [2,3,4,5] but the pointer will be in the 1th element of the new mylist ([2,3,4,5]) that is 3. and it will remove 3.so 2 will not be remove from the list.

thats why after the full operation [2,4] will be left.

using for loop:-
>>>for i in range(len(mylist)):
       mylist.remove(mylist[0])
       i-=1

>>>mylist
[]

you can do this using while loop:

>>>mylist = [1, 2, 3, 4, 5]
>>>while (len(mylist)>0):
       mylist.remove(mylist[0])
       print mylist

score 0 · Answer 6 · answered May 11 '13 at 11:32

0

Use a deep copy of mylist

mylist = [1, 2, 3, 4, 5]

l = [k for k in mylist] # deep copy

for i in range(len(mylist)):
    mylist.remove(l[i])
    print 

print mylist

answered May 11 '13 at 11:32

kiriloff

25,609
37
148
229

Why is list.remove only removing every second item?

6 Answers6

Linked