Using nested generator expression in Python 2.7

Question

In an appengine application, I want to build a set of all property names for a list of objects. This should be fairly straightforward:

users = security.User.all().fetch(1000)
props = set([k for k in u.properties().keys() for u in users])

However, the code above results in an error:

File "/Users/paulkorzhyk/Projects/appengine-flask-template/app/app.py", line 70, in allusers
props = set([k for k in u.properties().keys() for u in users])
UnboundLocalError: local variable 'u' referenced before assignment

After some experiments in the debugger I've noticed that adding a dummy expression fixes the code:

users = security.User.all().fetch(1000)
[u.properties().keys() for u in users]
props = set([k for k in u.properties().keys() for u in users])

This is quite counter-intuitive to me, why is original version failing in Python 2.7? and why adding a 'useless' expression in the middle fixes the problem?

As per this answer http://stackoverflow.com/questions/8049798/understanding-nested-list-comprehension the association should be left to right and hence reordering the loop statements should rectify. — Ifthikhan, Dec 26 '12 at 07:19

Abhijit · Accepted Answer · 2012-12-26T07:21:25.663

Just change the order of evaluation

props = set([k for k in u.properties().keys() for u in users])

to

props = set([k for u in users for k in u.properties().keys() ])

also you don't need a list comprehension but generator expression with a set comprehension would work here

props = set(k for u in users for k in u.properties().keys() )

Order of evaluation is from right to left

In your original expression

set([k for k in u.properties().keys() for u in users])

can be broken as

for k in u.properties().keys(): # Here u is undefined
    for u in users:
        #what ever

The interesting phenomena of using a Dummy expression is the fact that List Comprehension Leaks Variables, which causes u to be leaked in the global scope

So

[u.properties().keys() for u in users]

leaks u in global scope,

which makes

set([k for k in u.properties().keys() for u in users])

legitimate

The following example shows, how list comprehension leaks variables

>>> del i
>>> foo = [range(1,10) for _ in range(10)]
>>> globals()['i']

Traceback (most recent call last):
  File "<pyshell#84>", line 1, in <module>
    globals()['i']
KeyError: 'i'
>>> [i for i in foo]
[[1, 2, 3, 4, 5, 6, 7, 8, 9], [1, 2, 3, 4, 5, 6, 7, 8, 9], [1, 2, 3, 4, 5, 6, 7, 8, 9], [1, 2, 3, 4, 5, 6, 7, 8, 9], [1, 2, 3, 4, 5, 6, 7, 8, 9], [1, 2, 3, 4, 5, 6, 7, 8, 9], [1, 2, 3, 4, 5, 6, 7, 8, 9], [1, 2, 3, 4, 5, 6, 7, 8, 9], [1, 2, 3, 4, 5, 6, 7, 8, 9], [1, 2, 3, 4, 5, 6, 7, 8, 9]]
>>> globals()['i']
[1, 2, 3, 4, 5, 6, 7, 8, 9]
>>>

Thanks for the explanation, really helpful. Is this leaking of variables a bug? — Boycott Russia, Dec 26 '12 at 07:51
This is not a Bug but a Feature post Python 2.3. This feature has been removed from Python 3.0. Refer http://docs.python.org/2/reference/expressions.html#id20 — Abhijit, Dec 26 '12 at 07:58

score 1 · Answer 2 · answered Dec 26 '12 at 07:09

The reason your original example fails is that you have the for clauses in the wrong order. The for clauses in list/generator comprehensions are in the same order they would be if you wrote the code out as nested for loops. That is, the leftmost one is the outermost, the rightmost is the innermost. Switch the order of the for clauses to make it work.

The reason the dummy expression changes the behavior is that the dummy expression is a list comprehension, and in Python 2, list comprehensions (unlike generator comprehensions) leak their loop variable to the enclosing scope. The leaked u allows your second example to run, but it isn't doing what you think it is, because your for clauses in the props = ... line are still in the wrong order. It's only looping over one value of u, namely the last u value from the dummy expression.

Using nested generator expression in Python 2.7

2 Answers2