Why does a lambda function change its behavior from in the function that generates it to the main?

Question

I thought I understood lambda notation (in python) for defining functions, but I'm seeing a strange behavior that I don't understand. I looked at What do (lambda) function closures capture? and saw something that I think is related but isn't the same.

I generate a list of functions inside a function. Inside that function, the behavior of the functions in the list is as expected. However, when it's returned to main(), items 1 through 3 of the list all behave as item 3. I say "behave as" because you can see that the functions aren't actually identical -- they're stored at different memory locations.

I put in some debugging lines that are commented, which help a little in seeing what the code is doing. The lambda function depends on an integer, and the value of the integer is as expected inside the function but is the same for list items 1 through 3 in main().

I'd really appreciate if someone could help me understand what's going on here. Thanks!

import numpy as np

def make_func_list(matA):
    size = matA.shape[0]
    vec = np.ones(size)
    lst_f = []
    lst_f.append( lambda x: sum(x) )
    for ii in xrange(1,size):
        lst_f.append( lambda x: np.dot(matA[ii], x) )
    print '-----------------'
    print '-  in function  -'
    for ii in xrange(size):
        print 'func_list', ii, lst_f[ii](vec)
        print lst_f[ii] # note that the memory addresses are different
        if ii > 0:
            print lst_f[ii].func_closure[0].cell_contents # the integer is different
    print '-----------------'
    return lst_f

if __name__ == "__main__":
    size = 4
    matA = np.reshape(np.arange(size**2),(size,size))
    vec = np.ones(size)
    print "### matA ###"
    print matA
    func_list = make_func_list(matA)
    print '================='
    print '=   in main()   ='
    for ii in xrange(len(matA)):
        print 'func_list', ii, func_list[ii](vec)
        print func_list[ii] # memory addresses are unchanged from in make_func_list()
        if ii > 0:
            print func_list[ii].func_closure[0].cell_contents # but the integer is the same for all ii
    print '================='

score 1 · Accepted Answer · answered Oct 26 '14 at 21:48

1

In these lines:

for ii in xrange(1,size):
    lst_f.append( lambda x: np.dot(matA[ii], x) )

the lambda object that is created stores references to the matA and ii variables, and when the lambda is executed, it evaluates np.dot(matA[ii], x) using the values that matA and ii have at the time the lambda is called, NOT at the time the lambda was created. Unfortunately, the lambdas are called after the for loop has finished running, at which point the value of ii will be the last value assigned to it (i.e., size-1), and so all of the lambdas will behave in the way that only the last lambda is expected to behave.

There are several ways to get around this. One way is to store the current value of ii in a scope that only the current lambda can access, and this can be done by creating and immediately calling another lambda wrapped around the "main" lambda:

for ii in xrange(1, size):
    lst_f.append( (lambda ii_tmp: lambda x: np.dot(matA[ii_tmp], x))(ii) )

answered Oct 26 '14 at 21:48

jwodder

54,758
12
108
124

But does your explanation explain why the behavior was as expected in the printing for loop in the make_func_list() function? The lambdas are defined in one for loop and called in another within the make_func_list() function, and when they're called inside that function the behavior is as expected. It's only surprising (stuck on the last value of ii) when returned to main(). – dslack Oct 27 '14 at 05:59
@dslack Both `for` loops in `make_func_list` use the same `ii` variable, as they're within the same scope. When examining `lst_f[ii]` in the second `for` loop, the value of `ii` is the same as when the lambda was created, and so the behavior is as expected. – jwodder Oct 27 '14 at 10:16
Ah, I see. The fact that it's _called_ ii is crucial. If the second for loop in the function loops over jj instead, we get the same 'bizarre' behavior. – dslack Oct 28 '14 at 12:50

score 1 · Answer 2 · answered Oct 26 '14 at 21:50

Each time through the loop, the lambda expression creates a function object which contains a reference to ii whose value will looked up when the function is called. Since all the lambdas are evaluated in the same scope, they all contain a reference to the same name ii. The last value assigned to that name comes from the last iteration of the loop, so although the function objects were created in different iterations of the loop, the value that each will see for ii is the same, namely the last value that was obtained from xrange(size).

One simple (but slightly awkward) way to avoid this is to pass the value ii as a default value of an argument to the function object, since default values are evaluated at definition time, not call time.

lst_f.append( lambda x, _ii=ii: np.dot(matA[_ii], x) )

Why does a lambda function change its behavior from in the function that generates it to the main?

2 Answers2