47

Possible Duplicate:
Why does += behave unexpectedly on lists?

I found an interesting "feature" of the python language today, that gave me much grief.

>>> a = [1, 2, 3]
>>> b = "lol"
>>> a = a + b 
TypeError: can only concatenate list (not "str") to list
>>> a += b
>>> a
[1, 2, 3, 'l', 'o', 'l']

How is that? I thought the two were meant to be equivalent! Even worse, this is the code that I had a hell of a time debugging

>>> a = [1, 2, 3]
>>> b = {'omg': 'noob', 'wtf' : 'bbq'}
>>> a = a + b
TypeError: can only concatenate list (not "dict") to list
>>> a += b
>>> a
[1, 2, 3, 'omg', 'wtf']

WTF! I had lists and dicts within my code, and was wondering how the hell I ended up appending the keys of my dict onto a list without ever calling .keys(). As it turns out, this is how.

I thought the two statements were meant to be equivalent. Even ignoring that, I can kind of understand the way you append strings onto lists (since strings are just character arrays) but dictionaries? Maybe if it appended a list of (key, value) tuples, but grabbing only the keys to add to the list seems completely arbitrary.

Does anyone know the logic behind this?

martineau
  • 119,623
  • 25
  • 170
  • 301
Li Haoyi
  • 15,330
  • 17
  • 80
  • 137
  • 3
    This question was closed as a duplicate, but I'm not sure it is - the answer may be the same, but the question seems different. – Mark Ransom Aug 05 '11 at 05:54
  • This question has been asked other times too, even in the last few days, in other forms. – agf Aug 05 '11 at 05:55
  • 5
    If it's in every python tutorial, I have not seen it, and i've been working with python for the last year, written a Compiler and a Website in it and have spent what must be far too much time reading up random blurb about the features and deficiencies of python. I honestly never read this, or expected this behavior. I do not think "Mutable Sequence" is an obvious place to look for the fact that += is a separate operator from +, and in skimming the page again i did not see it at all. Searching "+=" gave me nothing useful. I don't think this behavior is as obvious as you think! – Li Haoyi Aug 05 '11 at 07:00
  • 1
    Wow, this behaviour just cost me multiple hours debugging. Good to know, but now I'm left with the question: Why? Why on all the earth would python do that? – lakerz Feb 08 '16 at 17:59

2 Answers2

42

This is and always has been a problem with mutability in general, and operator overloading specifically. C++ is no better.

The expression a + b computes a new list from the objects bound to a and b, which are not modified. When you assign this back to a, you change the binding of one variable to point to the new value. It is expected that + is symmetrical, so you can't add a dict and a list.

The statement a += b modifies the existing list bound to a. Since it does not change the object identity, the changes are visible to all bindings to the object represented by a. The operator += is obviously not symmetrical, it is equivalent to list.extend, which iterates over the second operand. For dictionaries, this means listing the keys.

Discussion:

If an object doesn't implement +=, then Python will translate it into an equivalent statement using + and =. So the two are sometimes equivalent, depending on the type of the objects involved.

The benefit of a += that mutates the referand (as opposed to the operand value, which is a reference) is that the implementation can be more efficient without a corresponding increase in implementation complexity.

In other languages, you might use more obvious notation. For example, in a hypothetical version of Python with no operator overloading, you might see:

a = concat(a, b)

versus

a.extend(a, b)

The operator notation is really just shorthand for these.

Bonus:

Try it with other iterables too.

>>> a = [1,2,3]
>>> b = "abc"
>>> a + b
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
TypeError: can only concatenate list (not "str") to list
>>> a += b
>>> a
[1, 2, 3, 'a', 'b', 'c']

It's useful to be able to do this, because you can append a generator to a list with += and get the generator contents. It's unfortunate that it breaks compatibility with +, but oh well.

Dietrich Epp
  • 205,541
  • 37
  • 345
  • 415
  • 2
    I don't know what you mean by "other iterables", this was the first example in his question? – agf Aug 05 '11 at 05:53
  • 8
    Here's another related [pitfall](http://zephyrfalcon.org/labs/python_pitfalls.html): `t = ([],); t[0] += [2, 3]`. The second statement will raise an exception, but afterwards, `t` is `([2, 3],)` even so. – Lauritz V. Thaulow Aug 05 '11 at 07:28
  • C++ absolutely is better. In C++, if `x` is of a mutable type that supports `+` (such as `std::string`) then `x += expr` always has the same meaning as `x = x + expr`, except that it may be more efficient. Of course you can overload `+=` in your own classes to do anything, but none of the standard library types behave in the crazy way that Python's do. – benrg May 08 '22 at 05:34
  • @benrg: It is absolutely incorrect to say that in C++, `x += y` always has the same meaning as `x = x + y`. – Dietrich Epp May 08 '22 at 20:25
  • I gave an example where it has the same meaning; can you give an example where it has a different meaning? – benrg May 08 '22 at 21:01
  • They are separately defined, you are free to define them how you wish. Just like Python. In Python, they are also separate operators, which can both be independently defined how you wish. – Dietrich Epp May 08 '22 at 22:18
  • @benrg: This is not the right place for an extended discussion about C++, however. – Dietrich Epp May 08 '22 at 22:19
  • @benrg: In the standard library, the `std::atomic` type has `+=` defined, but not `+`. – Dietrich Epp May 08 '22 at 22:24
8

The reason behind this is because the python lists (a in your case) implement the __iadd__ method, which in turns calls the __iter__ method on the passed parameter.

The following code snippet illustrates this better:

class MyDict(dict):
    def __iter__(self):
        print "__iter__ was called"
        return super(MyDict, self).__iter__()


class MyList(list):
    def __iadd__(self, other):
        print "__iadd__ was called"
        return super(MyList, self).__iadd__(other)


a = MyList(['a', 'b', 'c'])
b = MyDict((('d1', 1), ('d2', 2), ('d3', 3)))

a += b

print a

The result is:

__iadd__ was called
__iter__ was called
['a', 'b', 'c', 'd2', 'd3', 'd1']

The python interpreter checks if an object implements the __iadd__ operation (+=) and only if it doesn't it will emulate it by doing a add operation followed by an assignment.

GaretJax
  • 7,462
  • 1
  • 38
  • 47