2

I had someone recently ask me recently about the faux pas of changing a list while iterating over it. They presented the following scenario (which I have now updated with a better example) as a possible use case when the behavior might be desirable:

>>> jersey_numbers = [4, 2, 3, 5, 1]  # list of places in a race
>>> for jersey_number in jersey_numbers:
        if jersey_number == 2:  # disqualify jersey number 2 for a false start
            t.remove(jersey_number)
>>> t
[4, 3, 5, 1]  # jersey number 3 is now in second place

Is this behavior regular enough to use in a use case such as this one?

Justin O Barber
  • 11,291
  • 2
  • 40
  • 45
  • After `del` it skips the next value. – f p Jan 25 '13 at 18:23
  • possible duplicate of [Remove all occurences of a value from a Python list](http://stackoverflow.com/questions/1157106/remove-all-occurences-of-a-value-from-a-python-list) – Barmar Jan 25 '13 at 18:38
  • Possible duplicate of [Remove items from a list while iterating in Python](http://stackoverflow.com/questions/1207406/remove-items-from-a-list-while-iterating-in-python) – Justin O Barber Jan 24 '16 at 01:56

6 Answers6

7

When you remove an item from a list, everything in the list shifts over ...

[1, 2, 3, 4, 5]
#remove   ^
[1, 2, 3, 5]

If you do this while iterating over the object, and if you have items that you want to remove adjacent to each other, then when you remove the first, the second will shift over to take it's place. The for loop will continue incrementing the position in the list where it's pulling a value from which causes it to skip the value that bumped over to take the place of the item you deleted.

Here's a reference -- Just so we all know this is well documented behavior :)

mgilson
  • 300,191
  • 65
  • 633
  • 696
5

What you should be using instead is:

t = filter(None, t)  # or
t = [x for x in t if x]

Or if your condition is actually more complicated:

t = filter(lambda x: x != something, t)  # or
t = [x for x in t if x != something]

Btw., remove removes first matching element, not necessarily the one your x currently points to, though in your, simple case behaviour is equivalent.

What happens is that while you iterate the list you remove elements and iterator does not know about it, let's say your list is [1,0,0,2]:

  • iterator is at 1, no change, next
  • iterator is at 0, you remove some 0 which is the first one, list changed size, and now iterator points to second zero, next
  • iterator points to 2, no change

In effect your first algorithm removes every second zero.

Your second algorithm should not work, if you say it does, perhaps you did not test it enough.

Dima Tisnek
  • 11,241
  • 4
  • 68
  • 120
4

The mistake is that you're modifying the list while iterating over it. This doesn't behave the way you're expecting (when you delete the current element, the next element gets skipped).

Here is how it can be done:

In [18]: t = [5.0, 5.0, 5.0, 4.0, 0.0, 5.0, 0.0, 3.0, 5.0, 5.0, 0.0, 0.0, 4.0, 5.0, 0.0, 5.0, 4.0, 5.0, 3.0, 3.0, 5.0, 5.0, 5.0, 5.0]

In [19]: t[:] = [val for val in t if val != 0]
NPE
  • 486,780
  • 108
  • 951
  • 1,012
  • 2
    It is allowed and the results are defined -- It's just not immediately obvious how it works to new people. – mgilson Jan 25 '13 at 18:21
  • 2
    @mgilson: Didn't know the behaviour was well-specified. Could you provide a reference? Thanks. – NPE Jan 25 '13 at 18:22
  • 2
    why t[:] = ... and not t = ... ? – Paul Seeb Jan 25 '13 at 18:28
  • @PaulSeeb: Doesn't change the list reference. It could be that there are other references to `t` in the code we've not been shown. Since we don't know for sure, I've chosen the more conservative approach. – NPE Jan 25 '13 at 18:30
  • @NPE Seems intuitive now that you have said it. I had no idea. Any docs I could read about it? – Paul Seeb Jan 25 '13 at 18:31
  • @PaulSeeb -- That's known as slice assignment. – mgilson Jan 25 '13 at 18:33
  • @PaulSeeb: There's at least one good explanation here on SO of the difference between `l = ...` and `l[:] = ...`, but I am not able to find it at the moment. Perhaps somebody else could help? – NPE Jan 25 '13 at 18:34
  • I think I am all set. I remember now that @mgilson reminded me it was called slice assignments. Editing the list without changing the reference – Paul Seeb Jan 25 '13 at 18:36
  • @Wooble: I don't get what you're saying. OP's code modifies the list instead of creating a new one, and so does my code. – NPE Jan 25 '13 at 18:36
  • 1
    @Wooble -- Using slice-assignment will allow OP to correctly modify the list in place -- There's no problem with this one. – mgilson Jan 25 '13 at 18:37
  • @NPE -- As a side note, there is no controversy that it doesn't work the way OP thinks -- The controversy is whether it's well defined by the standard :) – mgilson Jan 25 '13 at 18:38
  • @mgilson: Absolutely. There's also no controversy around what we think is controversial. :) – NPE Jan 25 '13 at 18:39
  • FOUND IT!!! :) http://docs.python.org/2/reference/compound_stmts.html#the-for-statement – mgilson Jan 25 '13 at 18:48
  • @NPE -- I knew I had seen it somewhere. This one is so common and so rarely has an actual pointer to the docs -- I was beginning to wonder if I was making it up. – mgilson Jan 25 '13 at 18:51
  • Here's one question about slice assignments: http://stackoverflow.com/q/10623302/10077 – Fred Larson Jan 25 '13 at 21:04
2

You should iterate over a shallow copy of the list, because you're modifying the list and this may result in some item being missed during iteration.

For what you're doing, filter() is a good choice.

filter(lambda x:x != 0.0,t)

Using shallow copy:

In [7]: t = [5.0, 5.0, 5.0, 4.0, 0.0, 5.0, 0.0, 3.0, 5.0, 5.0, 0.0, 0.0, 4.0, 5.0, 0.0, 5.0, 4.0, 5.0, 3.0, 3.0, 5.0, 5.0, 5.0, 5.0]

In [8]: for x in t[:]:   # a shallow copy of t
    if x==0.0:
        t.remove(x)

In [9]: 0.0 in t
Out[9]: False
Ashwini Chaudhary
  • 244,495
  • 58
  • 464
  • 504
  • A good choice if you're a haskell programmer stuck with Python for some reason? (I find the list comprehension much more Pythonic and more readable as well) – Wooble Jan 25 '13 at 18:27
0
t = [e for e in t if e != 0.0]
Emil Ivanov
  • 37,300
  • 12
  • 75
  • 90
  • I'm not the downvoter, but have you considered that this doesn't really answer the question which was only about the behaviour of modifying while iterating. – Veedrac Aug 22 '14 at 21:16
  • The question was edited 11 times, so my answer should've been for a previous version. – Emil Ivanov Aug 28 '14 at 08:39
0

As @mgilson said, when you removed the 0.0 value, the second 0.0 took its place and was missed.

You could iterate over a shallow copy, and remove it from the original:

 >>> t = [1, 2, 3, 3, 3, 4, 5]
 >>> for value in t[:]:
         if value == 3:
             t.remove(value)
 >>> t
 [1, 2, 4, 5]
Nitzle
  • 2,417
  • 2
  • 22
  • 23