Python list generation pitfall

Question

Attempt 1

>>> width = 3
>>> height = 2

>>> zeros = [[0]*width]*height
>>> print(zeros)
[[0, 0, 0], [0, 0, 0]]
>>> zeros[1][2] = "foo"
>>> print(zeros)
[[0, 0, 'foo'], [0, 0, 'foo']]

Expected result was [[0, 0, 0], [0, 0, 'foo']]. It seems that the * operator populated each row with references to the same object, so modifying zeros[1] also modified zeros[0].

Attempt 2

So now we create the initial list object using list comprehension instead:

>>> zeros = [[0 for i in range(width)] for j in range(height)]
>>> print(zeros)
[[0, 0, 0], [0, 0, 0]]
>>> zeros[1][2] = "foo"
>>> print(zeros)
[[0, 0, 0], [0, 0, 'foo']]

Great, this is the expected result.

The question is...

If the * operator populates each row with references to the same object, then why doesn't every element within each row refer to the same object?

That is, with a (partial) understanding of why Attempt 1 didn't produce the expected output, I would expect that if ANY element of zeros is modified, ALL elements would be changed, that is:

>>> zeros = [[0]*width]*height
>>> print(zeros)
[[0, 0, 0], [0, 0, 0]]
>>> zeros[1][2] = "foo"
>>> print(zeros)

is expected to return:

>>> print(zeros)
[['foo', 'foo', 'foo'], ['foo', 'foo', 'foo']]

but instead returns:

>>> print(zeros)
[[0, 0, 'foo'], [0, 0, 'foo']]

So why does the second * operator populate each row with references to the same object, but each of the elements WITHIN any given row are unique objects?

Answer

List of lists changes reflected across sublists unexpectedly

tl;dr:

Integers are immutable, so [0]*3 creates copies the elements of [0], not references. However [0,0,0] is an object, so [0,0,0]*2 creates a new reference to the original object.

You're changing the reference at the index not changing the referenced object at that index when assigning to a list index — Iain Shelvington, May 12 '21 at 01:35
@IainShelvington Can you please elaborate? It looks like you are referring to 3x separate concepts in your comment, and in each case I am not clear on what you are referring to: `1// changing the reference at the index`, `2// changing the referenced object at that index`, `3// assigning to a list index` — chrispalmo, May 12 '21 at 01:55
"If the * operator populates each row with references to the same object, then why doesn't every element within each row refer to the same object?" **they absolutely do**. You can check this for yourself. `zeros = [[0]*width]*height` then `[id(z) for zs in zeros for z in zs]` — juanpa.arrivillaga, May 12 '21 at 02:00
@chrispalmo "Integers are immutable, so [0]*3 creates copies the elements of [0], not references." **no absolutely incorrect**. It **never, ever creates a copy**. The *type* of the object is irrelevant. Why would it create a copy for immutable elements? that's *exactly backwards* of what you would expect a runtime to do. — juanpa.arrivillaga, May 12 '21 at 02:06
This is totally wrong. `[0, 0, 0]` is an object, a list object, `0` is an object, an `int` object, **everything in python is an object**. Read the edits I added to my answer — juanpa.arrivillaga, May 12 '21 at 02:11

juanpa.arrivillaga · Accepted Answer · 2021-05-12T02:07:40.907

The fundamental disconnect here is that you believe:

zeros[1][2] = "foo"

Is *mutating the element in the rows. It isn't. It is mutating the list at zeros[1], which happens to be the same list at zeros[0].

You can't mutate int objects. Although, they are, in fact, the same int objects:

>>> zeros = [[0]*2]*2
>>> zeros
[[0, 0], [0, 0]]
>>> [[id(x) for x in zs] for zs in zeros]
[[4413403792, 4413403792], [4413403792, 4413403792]]

So, using int objects might actually obscure this, since you might have heard that small integers are cached in CPython. They are always the same object. But consider a user-defined class (which is mutable):

>>> from dataclasses import dataclass
>>> @dataclass
... class Foo:
...     foo: int
...
>>> fs = [[Foo(0)]*2]*3
>>> fs
[[Foo(foo=0), Foo(foo=0)], [Foo(foo=0), Foo(foo=0)], [Foo(foo=0), Foo(foo=0)]]
>>> fs[0][0].foo = 8
>>> fs
[[Foo(foo=8), Foo(foo=8)], [Foo(foo=8), Foo(foo=8)], [Foo(foo=8), Foo(foo=8)]]

So note, with the zeros example, there is no method we can call:

zeros[0][0].some_method()

Such that the int object at zeros[0][0] would be mutated, as would happen in the equivalent case of fs[0][0].foo = 8, because int objects don't expose mutator methods. That is what immutable means by definition.