Weird behavior in python list concatenation

Question

I create a Python list as

>>> list1 = ['a', 'b', 'c']

and set

>>> list2 = list1

Now I perform two similar operations to list1 and list2

>>> list1 = list1 + [1, 2, 3]
>>> list1
['a', 'b', 'c', 1, 2, 3]
>>> list2
['a', 'b', 'c']

and

>>> list2 += [1,2,3]
>>> list1
['a', 'b', 'c', 1, 2, 3]
>>> list2
['a', 'b', 'c', 1, 2, 3]

But the results are different in both the cases. What is the reason for it?

Just to ask, Don't the operation on both list1 and list2 suppose to perform the same job? and Didn't it already do it? where is the difference? — Bharadwaj, Jun 29 '15 at 16:00

score 2 · Answer 1 · answered Jun 29 '15 at 15:56

The += operator in Python for lists, actually internally calls the list.extend() function, and hence the list is extended in place.

Whereas when we do + concatenation operator , a new list is created and returned, hence the actual list which was there in list1 is not changed, but instead list1 now points to a new list.

Łukasz Rogalski · Answer 2 · 2015-06-29T16:00:27.243

Based on Ned Batchelder's talk on Pycon 2015:

__iadd__ on list is implemented as extending actual instance (according to Ned it's non-documented behavior of CPython). After list1 = list2 both names refers to same instance - so extending an instance is visible under second name.

__add__ actually creates a new list based on two input lists.

As a proof, consider following code snippet:

import dis

def f1():
    list1 += [1,2,3]

def f2():
    list1 = list1 + [1,2,3]

dis.dis(f1)
dis.dis(f2)

And let's check output:

>>> dis.dis(f1)
  2           0 LOAD_FAST                0 (list1)
              3 LOAD_CONST               1 (1)
              6 LOAD_CONST               2 (2)
              9 LOAD_CONST               3 (3)
             12 BUILD_LIST               3
             15 INPLACE_ADD
             16 STORE_FAST               0 (list1)
             19 LOAD_CONST               0 (None)
             22 RETURN_VALUE
>>> dis.dis(f2)
  2           0 LOAD_FAST                0 (list1)
              3 LOAD_CONST               1 (1)
              6 LOAD_CONST               2 (2)
              9 LOAD_CONST               3 (3)
             12 BUILD_LIST               3
             15 BINARY_ADD
             16 STORE_FAST               0 (list1)
             19 LOAD_CONST               0 (None)
             22 RETURN_VALUE

As you can see, += uses INPLACE_ADD, where l1 + l2 does not.

score 1 · Accepted Answer · answered Jun 29 '15 at 15:58

The reason behind this is that += and + calls two different methods of the class, __iadd__ method and __add__ method.

From an API perspective, iadd is supposed to be used for modifying mutable objects in place (returning the object which was mutated) whereas add should return a new instance of something. For immutable objects, both methods return a new instance, but iadd will put the new instance in the current namespace with the same name that the old instance had. This is why

i = 1
i += 1

seems to increment i. In reality, you get a new integer and assign it "on top of" i -- losing one reference to the old integer. In this case, i += 1 is exactly the same as i = i + 1. But, with most mutable objects, it's a different story:

As a concrete example:

a = [1, 2, 3]
b = a
b += [1, 2, 3]
print a  #[1, 2, 3, 1, 2, 3]
print b  #[1, 2, 3, 1, 2, 3]

compared to:

a = [1, 2, 3]
b = a
b = b + [1, 2, 3]
print a #[1, 2, 3]
print b #[1, 2, 3, 1, 2, 3]

notice how in the first example, since b and a reference the same object, when I use += on b, it actually changes b (and a sees that change too -- After all, it's referencing the same list). In the second case however, when I do b = b + [1, 2, 3], this takes the list that b is referencing and concatenates it with a new list [1, 2, 3]. It then stores the concatenated list in the current namespace as b -- With no regard for what b was the line before.

score 1 · Answer 4 · answered Jun 29 '15 at 15:59

That's because you are assigning a new object to list1 with the first operation, while you are changing the original object you assigned to list2 with the second one.

If you check using id() the id of the objects assigned to your variables, it's easier to understand:

>>> list1 = ['a', 'b', 'c']
>>> id(list1)
4394813200
>>> list2 = list1
>>> id(list2)
4394813200 # same id
>>> list1 = list1 + [1, 2, 3]
>>> id(list1)
4394988392 # list1 now references another object
>>> list1
['a', 'b', 'c', 1, 2, 3]
>>> id(list2)
4394813200 # list2 still references the old one
>>> list2
['a', 'b', 'c']
>>> list2 += [1,2,3]
>>> id(list2)
4394813200 # list2 still references the old one
>>> list2
['a', 'b', 'c', 1, 2, 3]
>>> id(list1)
4394988392
>>> list1
['a', 'b', 'c', 1, 2, 3]

Weird behavior in python list concatenation

4 Answers4