Can someone explain to me why the first code block doesn't change the list, but the second does.
a = [1,2,3]
for el in a:
el += 5
This leaves a as [1,2,3]. That said if I run
a = [1,2,3]
for i in range(len(a)):
a[i] += 5
then a = [6,7,8]. My guess is that in the first, when looping over the elements el is a temporary variable, and not actually the thing that references that element in the list. Not sure why incrementing it doesn't effect the list though.
Python integers are not mutable, but lists are.
In the first case el references immutable integers, so += creates a new integer that only el refers to.
In the second case the list a is mutated directly, modifying its elements directly. a[0] still references an immutable integer, so += creates a new integer, but its reference is assigned directly to an element of the mutable list.
Here are examples showing the reference ids of list elements. In the first case, new integers are created, but the original list references are unchanged.
a = [1,2,3]
print [id(x) for x in a]
print a
for el in a:
el += 5 # creates new integer, but only temp name references it
print [id(x) for x in a] # references are not changed.
print a
Output
[36615248, 36615236, 36615224]
[1, 2, 3]
[36615248, 36615236, 36615224]
[1, 2, 3]
In the second case, the list references are updated:
a = [1,2,3]
print [id(x) for x in a]
print a
for i in range(len(a)):
a[i] += 5 # creates new integer, but updates mutable list
print [id(x) for x in a] # references are changed.
print a
Output
[36615248, 36615236, 36615224]
[1, 2, 3]
[36615188, 36615176, 36615164]
[6, 7, 8]
= (when the left-hand side is just an identifier) is purely a syntactic construct, which binds the name on the left to the object on the right.
All other assignments are short-hand for various method calls.
a[i] = 3 is short for a.__setitem__(i, 3)a += 3 is short for a = a.__iadd__(3)a[i] += 3 is short for a.__setitem__(i, a[i]+3)The end result of each method call depends on how type(a) implements the method being called. The method might mutate its caller, or it might return a new object.
I first wrote this as a comment but I'd like to expand it a bit, especially to add the tuple example.
Mark Tolonen's answer is correct (and upvoted) in that ordinary integers are immutable (cannot be changed) and lists are mutable (can have elements replaced), but does not mention another couple of key concepts, which show up in slightly scary examples:
Objects get bound to variables.
Ordinary variable assignment like x = 3 simply binds the object on the right—which may be constructed on the spot if needed—to the name on the left.
"In-place" operators like += attempt to invoke modifier functions, which allows mutable objects to capture them. For instance, if x is bound to a class instance, writing x += 3 will actually execute x.__iadd__(3), if x has an __iadd__.1 If not, it runs x = x + 3 instead, which invokes the __add__ operator: x = x.__add__(3). See the operator documentation for all the gory details. In this case, the objects involved—ordinary integers—don't have modifier functions:
>>> (3).__add__
<method-wrapper '__add__' of int object at 0x801c07f08>
>>> (3).__iadd__
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
AttributeError: 'int' object has no attribute '__iadd__'
so this particular twist is irrelevant for int, but it's worth remembering.
Indexed assignment, x[i] = expression, invokes the __setitem__ method. This is how mutable (modifiable) objects mutate themselves. A list object implements __setitem__:
>>> [].__setitem__
<method-wrapper '__setitem__' of list object at 0x8007767e8>
For completeness, I will note that it also implements __getitem__ to retrieve x[i].
Hence, when you write a[i] += 5, you wind up calling:
a.__setitem__(i, a.__getitem__(i) + 5)
which is how Python manages to add 5 to the i'th element of the list bound to a.
Here's the slightly scary example. A tuple object is not modifiable, but a list object is. If we embed a list into a tuple:
>>> l = [0]
>>> t = (l,)
we can then use t[0] to invoke t.__getitem__ and t.__setitem__. Meanwhile t[0] is bound to the same list object as l. This part is obvious enough:
>>> t
([0],)
>>> l.append(1)
>>> t
([0, 1],)
We modified l in place, so t[0], which names the same list as l, has been modified. But now:
>>> t[0] += [2]
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
TypeError: 'tuple' object does not support item assignment
>>> t
([0, 1, 2],)
What?! How did t change when we got an error telling us that t can't change?
The answer is, t didn't change, but the list (which we can also access via l) did change. Lists implement __iadd__, so the assignment:
t[0] += [2]
"means":
t.__setitem__(0, t.__getitem__(0).__iadd__([2]))
The __getitem__ accessed the list and the __iadd__ augmented the list:
>>> l
[0, 1, 2]
and then the t.__setitem__(0, ...) raised the TypeError, but the list was already augmented by then.
Note, by the way, the fact that tweaking the list object bound to l affects the tuple object bound to t, because t[0] is that list object. This—the fact that variables are bound to objects, and elements within data structures like tuples, lists, and dictionaries, can refer to other objects—is critical in reading and writing Python code. Understanding both the binding rules, and when objects get created, is the key to knowing why this is usually a bad idea:
def f(a=[]):
Specifically, the list object here is created at def time, i.e., just once. Any time someone adds to the list inside f, with a.append for instance, it keeps adding to that original list!
See also python: how binding works for (way too) much more about additional binding rules. :-)
1As Duncan points out in a comment on chepner's answer, after invoking __iadd__, the returned result is re-bound to the object. (All functions return a result; returning without an expression, or "falling off the end" of a function, is defined as returning None.)
Mutable objects should generally return themselves, and immutable objects don't need to implement __iadd__ in the first place since implementing a mutation on a supposedly-immutable object just seems odd. Nonetheless, we can abuse, and thereby expose, this behavior by writing a class that pretends to be immutable, but is not. Here's an example. This is not meant to be useful, just to illustrate one of the darker corners of Python.
"""
demonstration of iadd behavior
"""
from __future__ import print_function
class Oddity(object):
"""
A very odd class: like a singleton, but there can be
more than one of them. Each one is a list that just
accumulates more items. The __iadd___ (+=) operator
augments the item, then advances to the next instance.
Creating them is tricky as we want to create new ones
up until we "freeze" the class, then start re-using
the instances. We use a __new__ operator a la immutable
objects, plus a boolean in the class itself, even though
each instance is mutable.
"""
def __new__(cls):
if not hasattr(cls, 'frozen'):
cls.frozen = False
if cls.frozen:
whichone = cls.rotator
cls.rotator = (whichone + 1) % len(cls.instances)
return cls.instances[whichone]
self = object.__new__(cls)
if not hasattr(cls, 'instances'):
cls.instances = []
self.whichone = len(cls.instances)
self.values = []
cls.instances.append(self)
print('created', self)
return self
def __str__(self):
return '#{}, containing {}'.format(self.whichone, self.values)
def __iadd__(self, value):
print('iadd to', self)
self.values.append(value)
all_oddities = self.__class__.instances
nextone = (self.whichone + 1) % len(all_oddities)
return all_oddities[nextone]
@classmethod
def freeze(cls):
if not hasattr(cls, 'frozen'):
raise TypeError('need at least one instance to freeze')
cls.frozen = True
cls.rotator = 0
# Now make two instances, and freeze the rest so that
# we can cycle back and forth.
o0 = Oddity()
o1 = Oddity()
Oddity.freeze()
print('o0 is', o0)
o0 += 'first add to o0'
o0 += 'second add to o0'
o0 += 'third add to o0'
print('now o0 is', o0, 'and o1 is', o1)
print('the first object is', Oddity.instances[0])
Once we create the two objects and freeze the class, we invoke __iadd__ three times on o0, so in the end, o0 and o1 are actually both bound to the second object. The first object—findable only through the class's cls.instances field—has two items in its list-of-items.
As an exercise, try to predict what this will print out, before running it.
(Extra-advanced exercise: turn Oddity into a metaclass that can be applied to classes to turn them into freeze-able multi-singletons. [Is there a term for "thing that is like a singleton but allows N of them"?] See also Why Singletons Are Controversial.)
[Edit: this had been bugging me: it turns out there is a name for fixed-set-of-singletons. When the set has just two elements, it's a "doubleton". In Python, True and False are the doubletons comprising the bool type. The generalization to n objects is a multiton, often instantiated / used as a fixed—at least, after "freeze" time—hash table. Note that Python's doubleton boolean instances are immutable, as is Python's singleton None.]
In the first code block, e1 is a reference to one of the elements in a (depending on which iteration of the loop we're in).
When you do e1 += 5 (or more clearly in this case, e1 = e1 + 5), what you are doing is reassigning e1 to the value of e1 + 5. After this operation, e1 no longer points to the first element of the list, but the new variable we created (e1 + 5). The list remains unchanged.
In the second code block, you are assigning something directly to an element of a list, as opposed to temporary reference e1.
In the first example, el is a local variable, and it's completely unaware that it's part of a list.
In the second example, you are explicitly manipulating elements of the list.
The lesson here will be slightly more clear if we expand the += shorthand, and if we use a list of mutable objects instead of immutable integers:
Iterator pattern:
a = [[1], [2]]
for el in a:
el = [3]
After this loop, a is unchanged:
>>> a [[1], [2]]
Counting loop pattern:
for i in range(len(a)):
a[i] = [3]
After this loop, a is changed:
>>> a [[3], [3]]
On the other hand, if you mutate el itself instead of reassigning it:
a = [[1], [2]]
for el in a:
el[0] = el[0] + 5
Note that here we are not reassigning el, we are mutating it without reassignment.
The change will be visible through the list too:
>>> a [[6], [7]]
In case of your original example with a list of integers, this kind of mutation is not possible, because integers are immutable. So if you want to modify the values inside the list, you must use the counting loop approach.
