How does list-comprehension "create list" and "append elements" different than simple loop?

Question

While I was reading and searching about what exactly causes performance difference between loops and list comprehension (based on below simple test cases, list comprehension faster), I encountered following posts here in SO:

Simple test cases:

def f1():
    t = []
    for i in range(10000):
        t.append(i)

def f2():
    t = [i for i in range(10000)]

Why is a list comprehension so much faster than appending to a list?

Are list-comprehensions and functional functions faster than “for loops”?

What I understand from above posts that main differences as follows;

For loop builds a list but, list comprehension does not
For loop loads append method on each iteration but, list comprehension does not

Then I used disassembler to see the details and I saw the below steps for following block of code:

Code:

def f2():
    t = [i for i in range(10000)]
    
dis.dis(f2)

Disassembler result:

0 BUILD_LIST              
2 LOAD_FAST                
4 FOR_ITER                 
6 STORE_FAST               
8 LOAD_FAST                
10 LIST_APPEND        
12 JUMP_ABSOLUTE            
14 RETURN_VALUE

Based on Python doc; 0 BUILD_LIST creates list and 10 LIST_APPEND uses append method by contrast with above related posts:

LIST_APPEND(i)
    Calls list.append(TOS[-i], TOS). Used to implement list comprehensions.

BUILD_LIST(count)
    Works as BUILD_TUPLE, but creates a list.

I could not figure out what I am missing here. Is the way list comprehension builds and appends different than for loop, because anyway LIST_APPEND contains append method and BUILD_LIST creates a list? or maybe the reason for the performance difference is something else? Can someone please clarify this for me?

Thanks a lot!

I think if you *also* use `dis.dis` on the for-loop version, the differences will become apparent to you. — Karl Knechtel, Jul 12 '20 at 19:17
@KarlKnechtel you are right, actually I did, there were additional steps, such as GET_ITER and CALL_METHOD. But in this case, difference between them is not mainly loading `append` method in each iteration or creating list as it is mentioned in given posts in my questions, is that right? — Sercan, Jul 12 '20 at 19:22
`LIST_APPEND` can *directly* use the C implementation of the method; when the code contains an `.append` call, that method has to be looked up through the normal mechanisms. My `dis` results look completely different from yours, though. Are you on 2.x? — Karl Knechtel, Jul 12 '20 at 19:28
@KarlKnechtel I am on 3.8.2. So `LIST_APPEND` actually not calling `append` method but using its C implementation. And regular `append` method in my code loads the method on each iteration, and this is the difference, did I understand correctly? — Sercan, Jul 12 '20 at 19:39
@juanpa.arrivillaga thank you, I did not notice the difference, that makes sense now — Sercan, Jul 12 '20 at 20:02

Yam Mesicka · Answer 1 · 2020-07-12T19:38:17.260

I've tried a different approach:

from collections import Counter


def f1():
    t = []
    for i in range(10000):
        t.append(i)

def f2():
    t = [i for i in range(10000)]

    
f1i = Counter(i.opname for i in dis.get_instructions(f1))
f2i = Counter(i.opname for i in dis.get_instructions(f2))

print(f"Only in regular append: {f1i - f2i}")
print(f"Only in list comprehension: {f2i - f1i}")

The results are (Python 3.7.6):

Only in regular append: Counter({'LOAD_FAST': 2, 'BUILD_LIST': 1, 'STORE_FAST': 1, 'SETUP_LOOP': 1, 'FOR_ITER': 1, 'LOAD_METHOD': 1, 'CALL_METHOD': 1, 'POP_TOP': 1, 'JUMP_ABSOLUTE': 1, 'POP_BLOCK': 1})
Only in list comprehension: Counter({'LOAD_CONST': 2, 'MAKE_FUNCTION': 1, 'CALL_FUNCTION': 1})

You can see that the "regular" append uses LOAD_METHOD (for list.append), LOAD_FAST, CALL_METHOD and POP_TOP each iteration:

dis.dis(f1)

  5           0 BUILD_LIST               0
              2 STORE_FAST               0 (t)

  6           4 SETUP_LOOP              26 (to 32)
              6 LOAD_GLOBAL              0 (range)
              8 LOAD_CONST               1 (10000)
             10 CALL_FUNCTION            1
             12 GET_ITER
        >>   14 FOR_ITER                14 (to 30)
             16 STORE_FAST               1 (i)

  7          18 LOAD_FAST                0 (t)
             20 LOAD_METHOD              1 (append)
             22 LOAD_FAST                1 (i)
             24 CALL_METHOD              1
             26 POP_TOP
             28 JUMP_ABSOLUTE           14
        >>   30 POP_BLOCK
        >>   32 LOAD_CONST               0 (None)
             34 RETURN_VALUE

It is also recommended to keep in mind that the opcodes change from one version to another.

Thank you, both @KarlKnechtel comments and your explanation are clear and useful. I was confused because I also saw `append` method in `LIST_APPEND` description — Sercan, Jul 12 '20 at 19:47
But you aren't actually looking at the byt code for the comprehension — juanpa.arrivillaga, Jul 12 '20 at 19:56
@juanpa.arrivillaga Is the result of `dis.dis` not byte code? I have looked for an option `view byte code` like in intellij idea for java, but could not find it. After googling, I found `dis.dis` to see byte code. Any link or direction you can give me to see byte code? — Sercan, Jul 12 '20 at 20:13
The point is that the work done inside the comprehension is hidden behind (I think) `CALL_FUNCTION`. It becomes a separate chunk of bytecode that `dis` doesn't reach into (at least when used naively). — Karl Knechtel, Jul 12 '20 at 23:13

How does list-comprehension "create list" and "append elements" different than simple loop?

1 Answers1