2

Basically, in the example below, why do the lambdas in f2 capture a reference to the list comprehension loop variable, but those in f3 "capture" (really, they just accept an argument) "the way we expect" lambda capture to work?

f1 = [lambda x: x, lambda x: x + 1, lambda x: x + 2]
print([f(0) for f in f1])
f2 = [lambda x: f(x) + 1 for f in f1]
print([f(0) for f in f2])
f3 = [(lambda ff: lambda x: ff(x) + 1)(f) for f in f1]
print([f(0) for f in f3])

The three lines outputted are:

[0, 1, 2]
[3, 3, 3]
[1, 2, 3]
wim
  • 338,267
  • 99
  • 616
  • 750
VF1
  • 1,594
  • 2
  • 11
  • 31

3 Answers3

4

f is the same variable throughout the list comprehension. A simpler example:

>>> fs = [lambda: x for x in [1, 2]]
>>> fs[0]()
2

One common hack to work around this without the extra lambda wrapper is to use a default (as defaults are evaluated when the function is), but that can be confusing to those new to the idiom and you should generally split it out into a new function instead.

>>> fs = [lambda x=x: x for x in [1, 2]]
>>> fs[0]()
1
Ry-
  • 218,210
  • 55
  • 464
  • 476
  • This doesn't really answer my question. I'm aware that the comprehension has the same variable - I'm asking why `f3` "works". – VF1 Sep 28 '17 at 21:37
  • @VF1: In `f3`, `ff` is not the same variable each time. It’s not even a variable in the same function – you have as many `(lambda ff: …)`s as elements in `f1`, and each time they’re called they create a scope with a `ff`, which becomes part of the `lambda x: ff(x) + 1` closure. In other words: `f2` has one variable `f` shared by every `lambda`; `f3` has *n* variables `ff`, each associated with a different `lambda`. – Ry- Sep 28 '17 at 21:48
3

In the second example, f is a free variable in the lambdas in f2 which is bound into their enclosing scope (the list comprehension). The functions take f's value at the time they are called. The functions in f2 are called in the following print statement, at which point f is already the third lambda from f1. Note that f is never in the scope in which f1, f2 and f3 are defined (presumably global or some broader function scope), but the lambdas always reference their enclosing scope which contains f.

In contrast, ff is a parameter in the outer lambdas f3, but it is bound in the call to those outer lambdas, which happens in the list comprehension that defines f3.

Sam Hartman
  • 6,210
  • 3
  • 23
  • 40
2

Python closures always capture variables, not objects.

In f2,

f2 = [lambda x: f(x) + 1 for f in f1]

all lambdas in the list capture the f variable, not the object the variable refers to at the time the lambda is defined. At the end of the comprehension, f refers to the last function in f1, so all lambdas in f2 find that function when they look up f.

In f3

f3 = [(lambda ff: lambda x: ff(x) + 1)(f) for f in f1]

each call to (lambda ff: lambda x: ff(x) + 1) creates a new local ff variable, and each lambda x: ff(x) + 1 captures a different ff variable. Unlike f, the ff variables are never reassigned, so the lambda x: ff(x) + 1 functions each see a different value of ff.

user2357112
  • 260,549
  • 28
  • 431
  • 505
  • This object-variable was the distinction I was looking for -- are there any relevant docs I could read on this? It seems that the argument `(f)` in the `f3` example gets passed by "object value", so then `ff` becomes a variable with the corresponding object value from `f1`. – VF1 Sep 28 '17 at 21:56
  • @VF1: There are the [Python execution model docs](https://docs.python.org/3/reference/executionmodel.html), and the [original closure PEP](https://www.python.org/dev/peps/pep-0227/). [Ned Batchelder's writeup on Python variable and assignment semantics](https://nedbatchelder.com/text/names.html) may also be useful, particularly the part about how function arguments are passed. – user2357112 Sep 28 '17 at 22:04