Strange behavior with nested list comprehension

Question

I have the following piece of code:

[e for e in [sl] for sl in [1,[2,3],4,5]]

which I thought being equivalent (in terms of output) to:

[sl for sl in [1,[2,3],4,5]]

Yet, while the latter produces: [1,[2,3],4,5] the former returns: [5, 5, 5, 5]

I think it must have something do with how nested for-statements are evaluated.

I found a similar case here Weird behavior: Lambda inside list comprehension but since it uses an anonymous functions, the reason behind this behavior should be different.

Clearly, there's something I'm missing and I don't see.

Thank you for any clarification

UPDATE

As Patrick pointed out, the order of the two for is wrong and shouldn't run unless sl was defined before. I fooled myself here because I ran the examples in the interpreter and [sl for sl in [1,[2,3],4,5]] was executed first leaving sl set to the last value of the list in globals()

Now it would be great to understand how this is evaluated

[e for e in [sl] for sl in [1,[2,3],4,5]]

in order to produce [5, 5, 5, 5] in output.

Even with the `for`s in the correct order, it won't run without a prior definition of `s1`. — Karl Knechtel, Aug 18 '22 at 05:45
Voted to close as a typo on that basis, and taking into consideration the accepted answer. This question cannot properly help others. — Karl Knechtel, Aug 19 '22 at 02:57

Patrick Haugh · Accepted Answer · 2017-10-08T16:24:19.003

2

Is sl defined elsewhere in your code? Perhaps as 5? As written, your first example should not run, and does not run for me in Python 3.6. The correct way to write it would be

[e for sl in [1,[2,3],4,5] for e in [sl]]

Note that here sl is defined before it is used.

Edit:

Python reads list comprehensions left to right. When it gets to for e in [sl], it evaluates the expression [sl] based on what it already knows, without reading the rest of the line. You list comprehension is then something like

[e for e in [5] for sl in [1,[2,3],4,5]]

As there are four sl in [[1,[2,3],4,5]], you get 5 four times in the resultant list.

When writing list comprehensions, it's natural to write them from smallest to biggest

e for e in x for x in y for y in z #wrong

but you should actually write them from right to left, so that the interpreter recognizes the identifiers that you use in the nested comprehensions

e for y in z for x in y for e in x

This is no different from regular for loops:

for e in x:
    for x in y:
        for y in z:
            print(e)

is pretty obviously wrong, and list comprehensions are no different.

edited Oct 08 '17 at 16:24

answered Oct 08 '17 at 12:37

Patrick Haugh

59,226
13
88
96

Well no, it's just that line actually and I'm running the examples in the interpreter. But yes, you are right actually! I must have ran `[sl for sl in [1,[2,3],4,5]] ` before and so the `sl` remained set to the last value of the list in `globals()`. That explains also why the inverted `for` worked. However, with `sl` set to `5` in globals, how would you explain the behavior of `[e for e in [sl] for sl in [1,[2,3],4,5]]`? – Daniele Bacarella Oct 08 '17 at 13:16
I mean, I understand that the leftmost `for` local `sl` would correctly override the global variable. I just want to understand how my solution is evaluated when `sl` is already set – Daniele Bacarella Oct 08 '17 at 13:22
@holak please see my edit (it got to big to fit in a comment). If that still doesn't help, feel free to ask more. – Patrick Haugh Oct 08 '17 at 16:25
1

@holak list comprehension variables "leaking" out of scope is a [known pseudo-bug in Python 2](https://stackoverflow.com/a/4199355/6779307) that is no longer present in Python 3. If there's no organizational pressure on you to be using Python 2, you should consider switching. – Patrick Haugh Oct 08 '17 at 16:34
1

list comprehension leaking the loop control variable is a nice catch - it caused the misunderstanding in the first place. No there's no org pressure, just playing around. Thanks! – Daniele Bacarella Oct 09 '17 at 15:12

score 0 · Answer 2 · answered Oct 08 '17 at 15:42

The only way that the code can run is if sl is defined elsewhere. If it is (as 5), then the code:

sl = 5

[e for e in [sl] for sl in [1,[2,3],4,5]]

produces the output of:

[5,5,5,5]

why?

The reason this is happening is that the for-loops evaluate from the left to the right. So the first thing is that happens is e is assigned to 5 - just as you could right:

[i for i in [9]]

which would give 9.

So now we know that regardless of the rightmost for-loop, the value of e will always be that of sl so in our case 5. Now, why is the output [5,5,5,5]? Well its confusing because the variable sl is being re-used. However, this does not effect the left of the list-comprehension as it evaluates left to right. So e will always have the value from [sl] (5) no matter what sl is on the right hand side. The right hand loop simply acts as a counter. Since there are 4 elements in it (1, [2,3], 4, 5), the left hand part is run 4 times. But e is always 5 so each of the times e is called, it is 5 - producing the result [5,5,5,5].

To demonstrate that the right hand side is simply a counter, the following will all produce the same result of [5,5,5,5]:

[e for e in [sl] for _  in [1, [2,3], 4, 5]]
[e for e in [sl] for sl in [0, 0, 0, 0]]
[e for e in [sl] for _  in range(4)]
[e for e in [sl] for sl in range(4)]

Strange behavior with nested list comprehension

2 Answers2