4

While upgrading code from Py2 to Py3, I ran into an oddity I can't quite explain. A list comprehension in a class definition can reference other class level variables in both python 2 and 3 as part of the for clause. However, if the variables are part of the if clause they throw a NameError in python 3 but work just fine in python 2.

I found a few related questions but they don't quite seem to explain the issue. Dictionary class attribute that refers to other class attributes in the definition is a similar issue in lambdas, but doesn't seem to be python version dependant. Additonally Why am I getting a NameError in list comprehension (Python)?, but has to do with scope in a debugger rather than a class.

The following code works fine on python 2.7.16, but fails on Python 3.7.4:

class A(object):
    a = [1, 2, 3]
    b = [n for n in a if n in a]

print(A.b)

In python 2 I get:

[1, 2, 3]

In python 3 I get:

Traceback (most recent call last):
  File "list_comprehension_closure.py", line 3, in <module>
    class A(object):
  File "list_comprehension_closure.py", line 5, in A
    b = [n for n in a if n in a]
  File "list_comprehension_closure.py", line 5, in <listcomp>
    b = [n for n in a if n in a]
NameError: name 'a' is not defined

However, the following works fine in both python 2 and 3 and the only difference is the if clause:

class A(object):
    a = [1, 2, 3]
    b = [n for n in a]

print(A.b)

Additionally, the following works in both python 2 and 3 and the only difference is that the comprehension is defined outside of a class block:

a = [1, 2, 3]
b = [n for n in a if n in a]

print(b)

I know there were some changes to closures and list comprehensions in Python 3, but I am not seeing anything that explains this difference.

Edit: For clarity, I am not looking for a work around. As demonstrated by my last example, I'm aware that moving the variables outside of the class scope I can resolve the issue. However, what I am looking for is an explanation for why this behavior was changed in python 3.

jbryan
  • 101
  • 5
  • Copy the python 2 code to your python 3 file and run it again – U13-Forward Aug 29 '19 at 02:03
  • @U10-Forward While it would have surprised me to find a significant change in either version of python in the 10 minutes since posting, for the sake of being a good sport I have run identical code in python 2 and 3 again and got the same result: ``` > cat list_comprehension_closure.py class A(object): a = [1, 2, 3] b = [n for n in a if n in a] print(A.b) > python2.7 list_comprehension_closure.py [1, 2, 3] > python3.7 list_comprehension_closure.py ... NameError: name 'a' is not defined ``` – jbryan Aug 29 '19 at 02:13
  • @hunzter Thanks, I didn't see that question in my search. However, it doesn't quite address the issue. The explanation makes sense as to why the first example where I reference `a` in the if clause fails. I may be missing something in the bytecode decomposition, but my reading of that would seem to suggest that my second example where `a` is referenced only in the `for` clause *should fail* in python 3, but it doesn't. – jbryan Aug 29 '19 at 02:33
  • @hunzter, actually on more careful reading, the linked question does address why my second example works. "There's one part of a comprehension or generator expression that executes in the surrounding scope, regardless of Python version. That would be the expression for the outermost iterable." – jbryan Aug 29 '19 at 02:40

1 Answers1

2

To me this look like an issue of inappropriate scope for the variable a. The following works by putting a in the higher scope:

a = [1, 2, 3]
class A(object):
    b = [n for n in a if n in a]

print(A.b)
  • I understand there are ways around this, and in fact for the sake of getting the code working I took a variant of the first approach you are suggesting. Your second approach does not work in either python version as `A` is not finished being defined when it is referenced. In both python 2 and 3 it throws `NameError: name 'A' is not defined` However, I am trying to understand the *why* of what changed between python 2 and 3 that would explain the difference in behavior. – jbryan Aug 29 '19 at 02:21
  • 1
    I removed the second part since you're right, it doesn't work. I must not have cleared everything first. I don't have an answer for why. –  Aug 29 '19 at 02:23