Undefined global in list generator expression using python3, works with python2, what changes are needed?

Question

class Some(object):
    tokens = [ ... list of strings ... ]
    untokenized = [tokens.index(a) for a in [... some other list of strings ...]]
    ... etc ...
some = Some()

This works fine with Python2.7. However python3 says:

Traceback (most recent call last):
File "./test.py", line 17, in <module>
    class Some(object):
File "./test.py", line 42, in Some
    untokenized = [tokens.index(a) for a in [... some other list of strings ...]]
File "./test.py", line 42, in <listcomp>
    untokenized = [tokens.index(a) for a in [... some other list of strings ...]]
NameError: global name 'tokens' is not defined

Though I can work around the problem, I really would like to know what's difference here between Python2 and Python3. I've read python 2->3 changes documents, but I was not able to identify any description which is related to my problem. Also 2to3 tool does not complain anything in my code.

By the way, though I can't recall the situation now, but I had something similar with python2 only too (I haven't even tried this with 3), I thought this should work (within a class):

def some_method(self):
    return {a: eval("self." + a) for a in dir(self) if not a.startswith("_")}

However it causes python2 saying: NameError: name 'self' is not defined I haven't tried this with python3 yet, but for example this works:

[eval("self." + a) for a in dir(self) if not a.startswith("_")]

If I change the relevant part of the previous example to this one (ok the example itself is a bit stupid, but it shows my problem at least). Now I am very curious, why self seems not to be defined for this first example but it is for the second? It seems with dicts, I have similar problem that my original question is about, but with list generator expression it works, but not in python3. Hmmm ...

After my python2 -> 3 problem I mentioned this, since all of these seems to be about the problem that something is not defined according to python interpreter (and maybe the second part of my question is unrelated?). I feel quite confused now. Please enlighten me about my mistake (since I am sure I missed something of course).

Answer 1

As Wooble says, the issue is that classes don't have a lexical scope (actually, in either Python 2 or Python 3). Instead, they have a local namespace that does not constitute a scope. This means that expressions within the class definition have access to the content of the namespace:

class C:
    a = 2
    b = a + 2    # b = 4

but scopes introduced within the body of the class do not have access to its namespace:

class C:
    a = 2
    def foo(self):
        return a    # NameError: name 'a' is not defined, use return self.__class__.a

The difference between Python 2 and Python 3 is that in Python 2 list comprehensions do not introduce a new scope:

[a for a in range(3)]
print a    # prints 2

whereas in Python 3 they do:

[a for a in range(3)]
print(a)    # NameError: name 'a' is not defined

This was changed in Python 3 for a couple of reasons, including to make list comprehensions behave the same way as generator-expressions (genexps); (a for a in range(3)) has its own scope in both Python 2 and Python 3.

So, within the body of a class, a Python 2 genexp or a Python 3 listcomp or genexp introduces a new scope and therefore does not have access to the class-definition local namespace.

The way to give the genexp/listcomp access to names from the class-definition namespace is to introduce a new scope, using a function or a lambda:

class C:
    a = 2
    b = (lambda a=a: [a + i for i in range(3)])()

---

The eval issue

The issue with your eval example is that eval by default evaluates its argument in the local scope; because Python 2 list comprehensions have the above behaviour of sharing the enclosing scope the eval can access the method scope, but a genexp or Python 3 listcomp local scope only has whatever the compiler can tell is required from the enclosing scope (since a genexp/listcomp scope is a closure):

def bar(x):
    return list(eval('x') + x for i in range(3))
bar(5)  # returns [10, 10, 10]

def baz(x):
    return list(eval('x') for i in range(3))
baz(5)  # NameError: name 'x' is not defined

As Martijn says, instead of eval you should use getattr.