How to copy a variable into a lambda from a list?

Question

I am trying to generate a list of lambdas that I will later apply to an object, but when I try to do it via a comprehension or a loop over a list, the reference to the variable is kept, rather than the value itself. Let me illustrate.

Assume your object class is something like this:

class Object:
 def function(self, x):
  print(x)

So when you create the object and invoke it you get something like this:

o = Object()
o.function(0)
>>> 0

Now, if I manually construct my list of lambdas it would look like this:

lambdas = [
 lambda x: x.function(0),
 lambda x: x.function(1),
 lambda x: x.function(2)
]

Which I can then apply to my previously created object:

for l in lambdas:
 l(o)
>>> 0
>>> 1
>>> 2

However, when I generate the lambda list from another list, I only get the reference to the latest element of the list:

lambdas = [lambda x: x.function(i) for i in range(2)]
for l in lambdas:
 l(o)
>>> 2
>>> 2
>>> 2

On closer inspection I can see that each lambda has a different memory address, so they are NOT references to the same function.

So I can only assume that the lambda is keeping a reference to i which has a final value of 2 and therefore when invoked, it takes the value.

So my question is if its possible to set the value of the variable inside the lambda before invocation?

Note: The usa case for a list of lambdas is to pass to the agg function of a Pandas groupby on a DataFrame. I am not looking for a solution to the pandas problem, but curious about the general solution.

Answer 1

Generator Option

Just change lambdas to a generator instead of a list, this will cause it redefine i on every call:

lambdas = (lambda x: x.function(i) for i in range(2))
for l in lambdas:
    print(l(o))

Full code:

class Object:
    def function(self, x):
        print(x)
o = Object()
o.function(0) #manual call
lambdas = (lambda x: x.function(i) for i in range(2))
for l in lambdas:
    l(o)

Output:

0 #output from manual call
0 #output from generator
1 #output from generator

List Option

If you need a list for things like lambdas[0](o) you can send i to lambda each iteration by using i=i like so:

lambdas = [lambda x, i=i: x.function(i) for i in range(2)]

Example of second option:

class Object:
    def function(self, x):
        print(x)
o = Object()
lambdas = [lambda x, i=i: x.function(i) for i in range(2)] #notice the cahnge
for i in range(len(lambdas)):
    lambdas[i](o) #notice the change

Output:

0
1

Answer 2

What takes place is that in this expression, the "living" (nonlocal) i variable is used inside each lambda created. And at the end of the for loop, its value is the last value taken - which will be used when the lambdas are actually called.

lambdas = [lambda x: x.function(i) for i in range(2)]

The fix for that is to create an intermediary namespace which will "freeze" the nonlocal variable value at the time the lambda is created. This is usually done with another lambda:

 lambdas = [(lambda i: (lambda x: x.function(i)))(i) for i in range(2)]

So, bear with me - in the above expression, for each execution of the for i loop, a new, disposable lambda i is created and called imediatelly with the current value of the i used in the for. Inside it, this value is bound to a local i variable, that is unique to this disposable lambda i (in Python internal workings, it gets its own "cell"). This unique iis then used in the second, permanent, lambda x expression. Whenever that one is called, it will use the i value persisted in the outter lambda i call. The external lambda i then returns the lambda x expression as its result, but its nonlocal i is bound to the value used inside the lambda i, not the one used in the for i.

This is a common problem in Python, but can't be fixed because it is part of how the language works.

There is a shorter, and working, form to "freeze" the i from for i when each lambda i is created, that does not require an outer function scope: when a function is created, the values passed as default for its parameters are stored along with the function. Then, if one stores the current value of i as a default value, it won't change when the variable i itself does:

lambdas = [lambda x, i=i: x.function(i) for i in range(2)]

Here, in the lambda x, i=i: snippet, the value of i in the scope the lambda is created is stored as the default value for the parameter i, which works as a local (in contrast with a nonlocal) variable inside the lambda function itself.