3

In nested one line for loop in static context the second loop does not accept variable (here 'l')

Do I have to understand that?

class T():
    l = 3
    t1 = [(k,v) for k in range(l) for v in range(3)] # ok
    t2 = [(k,v) for k in range(l) for v in range(l)] 
    #                   error 'l' is not defined ^
Ren Tier
  • 31
  • 1
  • Don't initialize variables here, do that if the init . or do you want them to be class variables ? – azro Sep 10 '22 at 11:36
  • @TurePålsson: Yes, that's on point I think. Thanks a lot. (still have to read it carefully) ( 'Classic' nested loops work.) – Ren Tier Sep 10 '22 at 12:08
  • @azro:Yes, I want to have some more complex class variables (without instantiation) – Ren Tier Sep 10 '22 at 12:09
  • 1
    And again it's very disappointing how fast questions are closed ... It would be very interesting to see the full depth of reasons and a fix at C-code level. This is NOT a duplicate. **It asks if one should understand that** and ... first remains unanswered because closed faster as I could provide the reasons why the answer to this question is: **you don't need to understand that** as it is an easy work-around available ... and ... closing it prevents providing deeper reasons reaching down to a fix and eventual disadvantage of such a fix at C-code level. I want this question reopened. – Claudio Sep 10 '22 at 12:46
  • More infomation: [improper scope in list comprehension, when used in class declaration](https://github.com/python/cpython/issues/47942), [Rejected alternative proposals: Changing the scope rules for comprehensions](https://peps.python.org/pep-0572/#changing-the-scope-rules-for-comprehensions) – Mechanic Pig Sep 10 '22 at 13:03
  • I think that [Martijn's answer to the duplicate](https://stackoverflow.com/a/13913933/550094), especially the "The (small) exception" part, explains perfectly what is going on here and the reasons why this choice was made - thanks to having read it some time ago, I was able to recognize what is going on here immediately. So, without more details about what might seem lacking in his answer, I see no reason at all to reopen this question. – Thierry Lathuille Sep 10 '22 at 18:59

1 Answers1

2

Check bytecode with dis.dis:

>>> dis('''class T:
...     l = 3
...     t = [(i, j) for i in range(l) for j in range(l)]''')

For the list comprehension in the class, the generated bytecode is as follows:

Disassembly of <code object <listcomp> at 0x000002B84F04ED90, file "<dis>", line 3>:
  3           0 BUILD_LIST               0
              2 LOAD_FAST                0 (.0)     # load iterator of "range(l)" of the outer for loop
        >>    4 FOR_ITER                13 (to 32)
              6 STORE_FAST               1 (i)
              8 LOAD_GLOBAL              0 (range)
             10 LOAD_GLOBAL              1 (l)      # load "l" of the inner for loop
             12 CALL_FUNCTION            1
             14 GET_ITER
        >>   16 FOR_ITER                 6 (to 30)
             18 STORE_FAST               2 (j)
             20 LOAD_FAST                1 (i)
             22 LOAD_FAST                2 (j)
             24 BUILD_TUPLE              2
             26 LIST_APPEND              3
             28 JUMP_ABSOLUTE            8 (to 16)
        >>   30 JUMP_ABSOLUTE            2 (to 4)
        >>   32 RETURN_VALUE

Notice the two comments I marked, which correspond to the bytecode of load the iterator of range(l) in the outer for loop and the bytecode of load l in the inner for loop respectively. The obvious difference is that the iterator of range(l) in the outer for loop is passed into the list comprehension as a function parameter, while the inner range(l) needs to be loaded dynamically. This attempts to find them from the global namespace, because l is not in the global namespace but in the private namespace of the class T. Therefore, the list comprehension cannot find it, resulting in an error.

Some different from list comprehension in functions: A well-known thing is that the same code will not make errors in the definition body of the function. This time, check the bytecode of the function with the same definition body:

>>> dis('''def foo():
...     l = 3
...     t = [(i, j) for i in range(l) for j in range(l)]''')
Disassembly of <code object <listcomp> at 0x000002B850EF5A50, file "<dis>", line 3>:
  3           0 BUILD_LIST               0
              2 LOAD_FAST                0 (.0)     # load iterator of "range(l)" of the outer for loop
        >>    4 FOR_ITER                13 (to 32)
              6 STORE_FAST               1 (i)
              8 LOAD_GLOBAL              0 (range)
             10 LOAD_DEREF               0 (l)      # load "l" of the inner for loop
             12 CALL_FUNCTION            1
             14 GET_ITER
        >>   16 FOR_ITER                 6 (to 30)
             18 STORE_FAST               2 (j)
             20 LOAD_FAST                1 (i)
             22 LOAD_FAST                2 (j)
             24 BUILD_TUPLE              2
             26 LIST_APPEND              3
             28 JUMP_ABSOLUTE            8 (to 16)
        >>   30 JUMP_ABSOLUTE            2 (to 4)
        >>   32 RETURN_VALUE

Note that the loading method of l in the inner loop is different from that in the class definition body. Python treats l here as closure, the bytecode of the function is here:

Disassembly of <code object foo at 0x000002B850EF5FD0, file "<dis>", line 1>:
  2           0 LOAD_CONST               1 (3)
              2 STORE_DEREF              0 (l)

  3           4 LOAD_CLOSURE             0 (l)    # treats "l" as closure
              6 BUILD_TUPLE              1
              8 LOAD_CONST               2 (<code object <listcomp> at 0x000002B850EF5A50, file "<dis>", line 3>)
             10 LOAD_CONST               3 ('foo.<locals>.<listcomp>')
             12 MAKE_FUNCTION            8 (closure)
             14 LOAD_GLOBAL              0 (range)
             16 LOAD_DEREF               0 (l)
             18 CALL_FUNCTION            1
             20 GET_ITER
             22 CALL_FUNCTION            1
             24 STORE_FAST               0 (t)
             26 LOAD_CONST               0 (None)
             28 RETURN_VALUE
Mechanic Pig
  • 6,756
  • 3
  • 10
  • 31
  • Nice explanation of the why and the how it comes, but ... it does not give an answer to the core of the question. That it is as it is you see from the Error message. The core of the question is in my eyes: is there a deeper sense in it or is it a kind of bug Python developers have not yet detected or if detected not yet fixed? – Claudio Sep 10 '22 at 11:56
  • @Claudio I think it is not a bug, because the iterator of the outer loop will only be built and used once, while the inner loop will need to build them many times or even build them dynamically, which cannot be passed into the list comprehension in the form of function parameters, and the private namespace of the class is different from the namespace of the function (such as the function can be executed many times to create the namespace, while the construction of the class will only be executed once), So list comprehension can only try to get `l` from the global. – Mechanic Pig Sep 10 '22 at 12:01
  • You still explain how it comes. There is sure always a reason and sure also a workaround, but in my eyes it is a kind of bug because you do not get what you see and what should be obvious. If you need to consider how things work under the hood the advantage of a higher level language is gone. – Claudio Sep 10 '22 at 12:13