2

Excusing the fact that the idiom is unclear in terms of its syntax and the behavior it should have, how does the 'for for' idiom for flattening a list of iterables work?

For example, this works:

>>> a = [[0, 1], [2, 3], [4, 5]]
>>> f = [x for l in a for x in l]
>>> f
[0,1,2,3,4,5]

...but so does this (the variables are all the same!):

>>> a = [[0, 1], [2, 3], [4, 5]]
>>> f = [a for a in a for a in a]
>>> f
[0,1,2,3,4,5]

And it's generalizable; we can go n-deep by expanding the comprehension further. For example, here n = 3:

>>> a = [[[0, 1], [2, 3]], [[4, 5]]]
>>> f = [a for a in a for a in a for a in a] # We just add in another 'for a in a' and...presto!
>>> f
[0,1,2,3,4,5]

So, this is weird – the variable a is ostensibly occurring with multiple values in the same expression without any explicit indication of which scope it is in (i.e. there are multiple scopes implied, one for each dimension of the list) and the value of a is different within each. Note that the comprehension is equivalent to the following for loop:

>>> l = []
>>> for x in x:
...     for x in x:
......      for x in x:
.........       l.append(x)
>>> l
[0,1,2,3,4,5]

...More Explicitly:

>>> l = []
>>> for b in a:
...     for c in b:
......      for d in c:
.........       l.append(d)
>>> l
[0,1,2,3,4,5]

We can try and map out the logic into the comprehension. Here are two examples:

# Example 1
>>> [d for c in a for b in c for d in b]
[0,1,2,3,4,5]

# Example 2
>>> [d for b in a for c in b for d in c]
[0,1,2,3,4,5]

Now just because the variable mappings of the examples above work, that doesn't mean either mapping makes any sense. I inferred both of these, but I can't for the life of me tell you what the logic behind the each example is. I was hoping by uncovering a mapping it would be clear, but I don't think either of these are. And herein lies my question:

What is the logical mapping for the nested comprehensions above and how does it generalize to n-dimensional lists of iterables?

And, to demonstrate:

What is the intuitive mapping for an n = 4 instance?

Update

The second mapping example (above) is the correct one. It seems blindingly obvious now. One way to see this, suggested in the answer to the duplicate question (see comments), is as follows:

>>> forest = [[[0, 1], [2, 3]], [[4, 5]]]
>>> [leaf for tree in forest for branch in tree for leaf in branch]
[0, 1, 2, 3, 4, 5]

And the generalization to n = 4 would be:

>>> forests = [[[[0, 1]], [[2, 3]]], [[[4, 5]]], [[[6, 7]]]]
>>> [leaf for grove in forest for tree in grove for branch in tree for leaf in branch]
[0, 1, 2, 3, 4, 5, 6, 7]

Mental model expanded.

Greenstick
  • 8,632
  • 1
  • 24
  • 29
  • perhaps a question you should ask yourself is, is there _ever_ a scenario you would use a list comprehension where `n >= 4` ? – gold_cy Jan 30 '20 at 00:02
  • Dupe? --> [Explanation of how nested list comprehension works?](https://stackoverflow.com/q/20639180/674039) – wim Jan 30 '20 at 00:05
  • This isn't a question of what to use in practice, this a question of what's going on with the interpreter. That said, given that I don't have to care what I name the variables in the list comprehension, it's trivial to do multidimensional flattenings with this method – not that it's the way to go. – Greenstick Jan 30 '20 at 00:06
  • 1
    I don't understand your question at all. Where's the confusion? Do you understand how the full loops work? How the list comps work? How one corresponds to the other? Is the issue with shadowing? If you look at your `a/b/c/d` loop example you'll see that each level only has to reference the previous one, so shadowing `a` or `x` in each step will also work (since the interpreter can't care about the name of the iterable being iterated over once a loop has been initialized). – Andras Deak -- Слава Україні Jan 30 '20 at 00:07
  • @wim Try extending that logic in the accepted answer further, it quickly becomes unclear. – Greenstick Jan 30 '20 at 00:07
  • 1
    @AndrasDeak But how is the comprehension following the same logic as the loops? I get the loops, those are clear. Look at the variable ordering in the comprehensions, that (to me) is not. – Greenstick Jan 30 '20 at 00:08
  • 2
    The for loops *always* unroll in same order as in the comprehension, regardless of how nested it is. Which part is unclear? – wim Jan 30 '20 at 00:08
  • Post a working comprehension that exhibits this and I'll accept it! – Greenstick Jan 30 '20 at 00:09
  • What wim said. Your question is a duplicate of what he posted. Nested comprehensions follow the same order as full loops, period. – Andras Deak -- Слава Україні Jan 30 '20 at 00:09
  • I would agree, but `[d for d in c for c in b for b in a]` does not work. – Greenstick Jan 30 '20 at 00:10
  • 1
    Try unrolling that in nested for loops in the same order (`for d in c: for c in b: for b in a: ...`), and notice that `c` is an undefined name. – Andras Deak -- Слава Україні Jan 30 '20 at 00:11
  • Here is the working comprehension: http://dpaste.com/2MNG4N1 I don't really want to post an answer because the question is a dupe – wim Jan 30 '20 at 00:12
  • So this is what I'm not clear on. Can you write up an answer that makes this clear? – Greenstick Jan 30 '20 at 00:12
  • 2
    I already did, on the dupe... :) – wim Jan 30 '20 at 00:13
  • 1
    The reason it works is because when you write `for x in y:`, it only evaluates `y` once when the loop starts. So you can reuse the variable inside the loop without affecting the outer loop. – Barmar Jan 30 '20 at 00:18
  • 1
    Finally see it – mapping it over to the example by @wim `[leaf for tree in forest for branch in tree for leaf in branch]` – Greenstick Jan 30 '20 at 00:27
  • yes, you got it! – wim Jan 30 '20 at 00:36

0 Answers0