2

I am writing some scientific python code. At some point in the code I need to accept a list of callable objects (existing_functions), and then produce another list of callable objects (changed_functions) by injecting a change-of-variables in the original objects' __call__ methods. To me, the following seems like a nice abstraction.

import numpy as np
changed_functions = [lambda x: f(np.exp(x)) for f in existing_functions]

I worry about misusing lambda here. So I ran some simple tests involving lambda in loops and generator expressions. Those simple tests produced confounding results, which seem to involve variable-scope or pass-by-reference issues which I cannot decipher.

The code snippet below tries four different methods to define a list of anonymous functions using lambda. Only the first method (with a list literal) works as I would hope.

# attempt 1: use a list literal
funcs = [lambda: 1, lambda: 2]
print(funcs[0]())  # 1
print(funcs[1]())  # 2
# attempt 2: much closer to my actual use-case.
gunks = [lambda: i for i in [1,2]]
print(gunks[0]())  # 2, but I expect 1.
print(gunks[1]())  # 2, as expected.
# attempt 3: an ugly version of my use case, which I thought might work.
hunks = [(lambda: i,)[0] for i in [1,2]]
print(hunks[0]())  # 2, but I expect 1.
print(hunks[1]())  # 2, as expected.
# attempt 4: another ugly version of my use case. "WE NEED MORE LAMBDA."
yunks = [(lambda: lambda: i)() for i in [1,2]]
print(yunks[0]())  # 2, but I expect 1.
print(yunks[1]())  # 2, as expected.

I am hoping that someone can explain to me why the last three methods in the snippet above all produce such a different result, compared to a list-literal.

user2664946
  • 33
  • 4
  • `gunks = [lambda: i for i in [1,2]]` you say is closer to your intended use but I'm not sure I follow the logic. Can you explain the problem that you're trying to solve? You might want to look at [this](https://stackoverflow.com/questions/233673/how-do-lexical-closures-work) – roganjosh Jun 29 '19 at 20:43
  • the linked duplicate may not be exactly a duplicate, but i think the explanations in the answers there are the best. – juanpa.arrivillaga Jun 29 '19 at 21:04
  • @roganjosh the definition of ``gunks`` misses the reason of why I would do this (injecting a change of variables into a function definition), but it matches the syntax. For readability / clarity I probably should have defined actual functions beforehand (e.g. ``f1 = lambda x: 2*x``, ``f2 = lambda x: x**2``), but I didn't think that would affect the code behavior. – user2664946 Jun 29 '19 at 21:07

1 Answers1

3

You bind this with the i variable, but since you later change i, it will look for a variable with that name in the scope, and thus pick the last value you assigned to that i.

You can resolve this by adding an extra parameter to your lambda function:

changed_functions = [lambda i=i: i for i in [1,2]]

Or if you do not want that, you can use currying [wiki] here:

changed_functions = [(lambda j: lambda: j)(i) for i in [1, 2]]

or we can use an explicit function:

def function_maker(j):
    return lambda: j

changed_functions = [function_maker(i) for i in [1, 2]]

We here thus create a lambda-expression that takes a variable i and returns a lambda expression that returns that i. We then bind the i with our value for i, and thus construct an i in a more local scope.

for both it then returns:

>>> changed_functions[0]()
1
>>> changed_functions[1]()
2
Willem Van Onsem
  • 443,496
  • 30
  • 428
  • 555
  • 1
    `changed_functions = [(lambda i: lambda: i)(i) for i in [1, 2]]` is that legitimate Python? I mean, can that ever be reasonably expected in code bases? I've never seen anything like it. – roganjosh Jun 29 '19 at 20:44
  • @roganjosh: yes, I've tested it. It is however more common in functional languages like Haskell (since *eta-reductions* result in a less "noisy" language). – Willem Van Onsem Jun 29 '19 at 20:45
  • @roganjosh: see for example the *Y-combinator* that is used to perform recursion in lambda-calculus: https://en.wikipedia.org/wiki/Fixed-point_combinator#Fixed_point_combinators_in_lambda_calculus – Willem Van Onsem Jun 29 '19 at 20:46
  • 1
    @roganjosh In python one would normally just use a regular function definition to define the higher order function, something like `def func_maker(i): return lambda: i` and then `[func_maker(i) for i in whatever]` but it is equivalent to the lambda expression above (just more pythonic, in my opinion). – juanpa.arrivillaga Jun 29 '19 at 20:49
  • 1
    @juanpa.arrivillaga I think that approach makes it a bit easier for my brain to parse it out. Still, that's taught me something :) – roganjosh Jun 29 '19 at 20:52
  • 1
    @roganjosh and actually, the common idiom is to take advantage of default arguments, so `[lambda i=i:i for i in whatever]`, however, I personally think this is a bad solution, because now your function has a different signature than what you intended. If i want to amke a funtion that takes no arguments, I want it to complain loudly if I pass an argument to it, not silently change the output – juanpa.arrivillaga Jun 29 '19 at 20:53
  • 1
    Thank you so much! Although maybe ``changed_functions = [(lambda j: lambda: j)(i) for i in [1, 2]]`` would make the meaning of dummy variables clearer. – user2664946 Jun 29 '19 at 21:10
  • 1
    @user2664946 or *just use a full function definition to make it even more clear* – juanpa.arrivillaga Jun 29 '19 at 21:14
  • @juanpa.arrivillaga yes I agree the full function definition is better in practice. That is what I ended up doing in my application. However my question asked *why* python was behaving in the way it was. So an answer which sticks to lambdas is still relevant to me (especially in the context of the other question you linked, https://stackoverflow.com/questions/2295290/what-do-lambda-function-closures-capture). – user2664946 Jun 29 '19 at 21:20
  • @user2664946 it is important to understand, that lambdas and regularly defined functions would work *exactly the same* as far as this. – juanpa.arrivillaga Jun 29 '19 at 21:23