List concatenation using "+=" vs "extend" in Python 2 & Python 3

Question

I was doing some analysis for doing list concatenation using two different approaches for varying lengths of lists, with different Python versions.

Using += in Python 2:

$ python -m timeit -s "li1=range(10**4); li2=range(10**4)" "li1 += li2"  
10000 loops, best of 3: 55.3 usec per loop  
$ python -m timeit -s "li1=range(10**6); li2=range(10**6)" "li1 += li2"  
100 loops, best of 3: 9.81 msec per loop

Using += in Python 3:

$ python3 -m timeit -s "li1=list(range(10**4)); li2=list(range(10**4))" "`li1 += li2`"  
10000 loops, best of 3: 60 usec per loop  
$ python3 -m timeit -s "li1=list(range(10**6)); li2=list(range(10**6))" "`li1 += li2`"  
100 loops, best of 3: 11 msec per loop

Using extend in Python 2:

$ python -m timeit -s "li1=range(10**4); li2=range(10**4)" "li1.extend(li2)"  
10000 loops, best of 3: 59 usec per loop  
$ python -m timeit -s "li1=range(10**6); li2=range(10**6)" "li1.extend(li2)"  
100 loops, best of 3: 10.3 msec per loop

Using extend in Python 3:

$ python3 -m timeit -s "li1=list(range(10**4)); li2=list(range(10**4))" "li1.extend(li2)"  
10000 loops, best of 3: 64.1 usec per loop  
$ python3 -m timeit -s "li1=list(range(10**6)); li2=list(range(10**6))" "li1.extend(li2)"  
100 loops, best of 3: 11.3 msec per loop

I'm surprised to notice that Python 3 is much slower than Python 2, and the jump is huge.
Another quite interesting thing is doing += is much more faster than doing an extend.

Any implementation reasons to justify the above observations, especially between Python 2 & Python 3?

Check this answer out: http://stackoverflow.com/questions/3653298/concatenating-two-lists-difference-between-and-extend — abe, Feb 07 '16 at 01:47
A 10% performance change between versions like this is not "huge". Is the performance of `list.extend` really important to your code? I can't imagine how it could be, since almost any real task will be vastly slower than the list extending bits. If you're doing a lot of extending (relative to other stuff), you may be using the wrong algorithm. — Blckknght, Feb 07 '16 at 02:51
I was always under the impression that advanced versions of a programming language implementation(here Python 3) is always better than previous versions(Python 2.x). — kmario23, Feb 07 '16 at 15:02
The answer to your second sub-question is: `extend` involves an extra function call, which is marginally (not "much more"!) slower than `+=`, which can be optimized under the hood. See http://stackoverflow.com/questions/3653298/concatenating-two-lists-difference-between-and-extend — Ferdinand Beyer, Feb 07 '16 at 16:58

score 3 · Accepted Answer · edited May 23 '17 at 11:45

I'm surprised to notice that Python 3 is much slower than Python 2, and the jump is huge.

As others have pointed out, your benchmarks don't support your statement that Python 3 is much slower. 9.81 msec vs 11 msec, 10.3 msec vs 11.3 msec?

It is a documented fact that Python 3 is a bit slower than Python 2. The What's new document states on performance:

The net result of the 3.0 generalizations is that Python 3.0 runs the pystone benchmark around 10% slower than Python 2.5. Most likely the biggest cause is the removal of special-casing for small integers. There’s room for improvement, but it will happen after 3.0 is released!

Another quite interesting thing is doing += is much more faster than doing an extend.

Again, "much [more] faster" is exaggerated. 9.81 msec vs 10.3 msec?

For the answer, see this related SO question. extend requires a Python-level function call, vs += can be optimized at the C-level, so it is marginally faster.

List concatenation using "+=" vs "extend" in Python 2 & Python 3

1 Answers1