Explain this list comprehension oddity in the general case and what are it's use cases?

Question

Yep. This happend. When I absent mindedly put an index on a in variable. Explain (it?) away. What is happening in the general case and what are it's use cases?

>>> [q for q[0] in [range(10),range(10,-1,-1)]]

Traceback (most recent call last):
  File "<pyshell#209>", line 1, in <module>
    [q for q[0] in [range(10),range(10,-1,-1)]]
NameError: name 'q' is not defined

>>> [q for q in [range(10),range(10,-1,-1)]]
[[0, 1, 2, 3, 4, 5, 6, 7, 8, 9], [10, 9, 8, 7, 6, 5, 4, 3, 2, 1, 0]]
>>> [q for q[0] in [range(10),range(10,-1,-1)]]
[[[10, 9, 8, 7, 6, 5, 4, 3, 2, 1, 0], 9, 8, 7, 6, 5, 4, 3, 2, 1, 0], [[10, 9, 8, 7, 6, 5, 4, 3, 2, 1, 0], 9, 8, 7, 6, 5, 4, 3, 2, 1, 0]]
>>> [q for q[0] in [range(10),range(10,-1,-1)]]
[[[10, 9, 8, 7, 6, 5, 4, 3, 2, 1, 0], 9, 8, 7, 6, 5, 4, 3, 2, 1, 0], [[10, 9, 8, 7, 6, 5, 4, 3, 2, 1, 0], 9, 8, 7, 6, 5, 4, 3, 2, 1, 0]]
>>> [q for q[0] in [range(10,-1,-1),range(10)]]
[[[0, 1, 2, 3, 4, 5, 6, 7, 8, 9], 9, 8, 7, 6, 5, 4, 3, 2, 1, 0], [[0, 1, 2, 3, 4, 5, 6, 7, 8, 9], 9, 8, 7, 6, 5, 4, 3, 2, 1, 0]]
>>> [q for q[0] in [range(10),range(10,-1,-1)]]
[[[10, 9, 8, 7, 6, 5, 4, 3, 2, 1, 0], 9, 8, 7, 6, 5, 4, 3, 2, 1, 0], [[10, 9, 8, 7, 6, 5, 4, 3, 2, 1, 0], 9, 8, 7, 6, 5, 4, 3, 2, 1, 0]]
>>> [q for q[0] in [range(10,-1,-1),range(10)]]
[[[0, 1, 2, 3, 4, 5, 6, 7, 8, 9], 9, 8, 7, 6, 5, 4, 3, 2, 1, 0], [[0, 1, 2, 3, 4, 5, 6, 7, 8, 9], 9, 8, 7, 6, 5, 4, 3, 2, 1, 0]]
>>> [q for q[0] in [range(10),range(10,-1,-1)]]
[[[10, 9, 8, 7, 6, 5, 4, 3, 2, 1, 0], 9, 8, 7, 6, 5, 4, 3, 2, 1, 0], [[10, 9, 8, 7, 6, 5, 4, 3, 2, 1, 0], 9, 8, 7, 6, 5, 4, 3, 2, 1, 0]]
>>> [q for q[0] in [range(10),range(10,-1,-1)]]
[[[10, 9, 8, 7, 6, 5, 4, 3, 2, 1, 0], 9, 8, 7, 6, 5, 4, 3, 2, 1, 0], [[10, 9, 8, 7, 6, 5, 4, 3, 2, 1, 0], 9, 8, 7, 6, 5, 4, 3, 2, 1, 0]]
>>> [q for q in [range(10),range(10,-1,-1)]]
[[0, 1, 2, 3, 4, 5, 6, 7, 8, 9], [10, 9, 8, 7, 6, 5, 4, 3, 2, 1, 0]]

@kratenko: being able to use `q[0]` as the loop target name and have it work, and what happens when you do so. — Martijn Pieters, May 23 '14 at 15:00
@freegnu, it appears you have introduced a variable named `q` in your scope after running your first line and before running your third. — Frédéric Hamidi, May 23 '14 at 15:02
@MartijnPieters as far as I can see it did not work, and what happens is a `NameError` which sounds the thing that should happen, if you try to access `q` without defining it prior. — kratenko, May 23 '14 at 15:03
@kratenko: I know, which is why I wrote that as part of my answer. — Martijn Pieters, May 23 '14 at 15:07
@IanAuld: that's because you a) would have to use `list(range(...))` and b) would have to set `q = list(range(...))` first in Python 3. — Martijn Pieters, May 23 '14 at 15:18
Note that your question is *ambiguous*. Some people interpreted your question to be about why `q` first raises a `NameError` but then doesn't, while I focused on the item-assignment aspect (`q` vs. `q[0]` as a loop target). Can you please clarify *what exactly* about the output puzzled you? — Martijn Pieters, May 23 '14 at 15:24
@MartijnPieters I was asking about the undocumented indexed assignment in a list comprehension and why it seems to do an assignment of the whole list to the source lists before the first loop and then return the contents of that assignment for both/all lists. — freegnu, May 23 '14 at 15:59
@freegnu: Right, so it *is* about index assignment (which *is* documented, see my answer). This is not limited to a list comprehension, but works for *all* `for` loops, because it works for *assignments*. — Martijn Pieters, May 23 '14 at 16:07
@freegnu: can you update your question to include that? That's important information that should not be lost in the comments. — Martijn Pieters, May 23 '14 at 16:08
@MartijnPieters Although that is an interesting question in and of itself it is not really the question I am asking or would have asked. The question I'm asking is what is going on here? Which was already answered. I'm a little disappointed about the mundaneness of the actual explaination as I was hoping to find a hidden gem not a glaring problem with the Python 2.x language implementation leaking variables out of list comprehensions before 3.x. I can find those easily enough in Ruby without even trying. — freegnu, May 23 '14 at 16:14
freegnu: To be fair, it was a deliberate decision; a regular `for` loop also executes in the current scope. It was only when generator expressions were added too that it became clear a separate scope was called for and made sense. — Martijn Pieters, May 23 '14 at 16:16
@MartijnPieters Sorry, not trying to be harsh on you. Probably more upset with myself because I vaguely remember having dealt with the reality of list comprehension variable leakage years ago and have been subconsciously ignoring the signs of it for years now. — freegnu, May 23 '14 at 16:24

score 5 · Accepted Answer · edited May 23 '17 at 12:05

The for loop compound statement reuses the target_list construct also used in assignments:

for_stmt ::=  "for" target_list "in" expression_list ":" suite

That's because for each iteration of the loop, the 'current' value is assigned to the target list. That also means you can assign to an item. Quoting the documentation:

Each item in turn is assigned to the target list using the standard rules for assignments,

In Python, this is perfectly normal:

q = [None]
q[0] = 'foobar'

and you can do the same in a for loop, provided the name q exists and supports item assignment. This is why your first attempt failed:

>>> [q for q[0] in [range(10),range(10,-1,-1)]]
Traceback (most recent call last):
  File "<pyshell#209>", line 1, in <module>
    [q for q[0] in [range(10),range(10,-1,-1)]]
NameError: name 'q' is not defined

There is no name q yet, so assigning to q[0] fails.

There probably is no real use case for using this in a for loop, but the advantage of reusing target_list here is that it makes the grammar and parser simpler, and lets you use assignment unpacking, e.g. assignment to multiple targets:

for key, value in mapping.items():
    # unpack two-value tuples returned by .items() into two names

Next, you executed a straight-up list comprehension:

>>> [q for q in [range(10),range(10,-1,-1)]]
[[0, 1, 2, 3, 4, 5, 6, 7, 8, 9], [10, 9, 8, 7, 6, 5, 4, 3, 2, 1, 0]]

In Python 2, the local names of a list comprehension leak; they live in the scope the list comprehension is defined in. See Python list comprehension rebind names even after scope of comprehension. Is this right?

So the moment the above list comprehension is executed, the name q exists, still bound to the value of the last iteration:

>>> q
[10, 9, 8, 7, 6, 5, 4, 3, 2, 1, 0]

This is a list object, so it supports item assignment. From there on out, what you do is item assignment to q[0] and re-referencing this one global q name:

>>> [q for q[0] in [range(10),range(10,-1,-1)]]
[[[10, 9, 8, 7, 6, 5, 4, 3, 2, 1, 0], 9, 8, 7, 6, 5, 4, 3, 2, 1, 0], [[10, 9, 8, 7, 6, 5, 4, 3, 2, 1, 0], 9, 8, 7, 6, 5, 4, 3, 2, 1, 0]]
>>> q
[[10, 9, 8, 7, 6, 5, 4, 3, 2, 1, 0], 9, 8, 7, 6, 5, 4, 3, 2, 1, 0]

You've put each of the two range()-produced lists in index 0 of q, but after the list comprehension is done, q[0] is bound to the last one being iterated over.

In Python 3, the item assignment in a for loop will still work, but list comprehensions no longer 'leak' local names as they are executed in a separate scope. Dict and set comprehensions, as well as generator expressions, are executed in a separate scope, always, regardless of Python versions.

score 1 · Answer 2 · answered May 23 '14 at 15:04

There is no oddity in the way list comprehension is working. You need to know what the contents of your variables are, between the sequence of operations you are doing.

The first line in your snippet

[q for q[0] in [range(10),range(10,-1,-1)]]

errors out because the interpreter does not know what q is yet. And you are trying to access the first element of an unknown variable by doing q[0]. You need to be doing:

[q[0] for q in [range(10),range(10,-1,-1)]]

As to why it works in the next sequence of operations?

After you do:

[q for q in [range(10),range(10,-1,-1)]]

q is now a variable with value [10, 9, 8, 7, 6, 5, 4, 3, 2, 1, 0] and hence your next operation:

[q for q[0] in [range(10),range(10,-1,-1)]]

does not fail. If in the above operation [q for q[0] in [range(10),range(10,-1,-1)]] you use another variable p instead of q, you would get the original error you saw the first time.

I have been working under the incorrect assumption that there was no leakage of list comprehension variables in Python for the last 13 years. I thought there was some hidden gem to be found in list comprehension bahavior. — freegnu, May 23 '14 at 16:07

Explain this list comprehension oddity in the general case and what are it's use cases?

2 Answers2