3

I'd like to understand why the two prints below produce different results:

f = [lambda x: x**2, lambda x: x+100]
f_new = [lambda x: fi(x) for fi in f]
print( [fi(2) for fi in f] )
print( [fi(2) for fi in f_new] )

The output is:

[4, 102]
[102, 102]
Max Alekseyev
  • 153
  • 5
  • 1
    Are you under the impression that `^` means "exponentiate" in Python? It doesn't; it's bitwise XOR. – user2357112 Feb 23 '21 at 21:46
  • @user2357112supportsMonica: Sorry, I copied code from Sage. I've just fixed it to make consistent with pure Python. – Max Alekseyev Feb 23 '21 at 21:48
  • 1
    While your example is complete, it's not what I'd call minimal. You'd get the same "unexpected" behavior from something as simple as `l = [lambda x: fi(x) for fi in f]`. I'd recommend editing-out all of the references to `itertools` since they're distracting from the question you're really asking – Brian61354270 Feb 23 '21 at 21:49
  • The itertools documentation has *roughly equivalent* functions for those two *methods* - were they any help in understanding how they were working? – wwii Feb 23 '21 at 21:50
  • @Brian: I seem to get it! I'll rewrite the question. – Max Alekseyev Feb 23 '21 at 21:55
  • The suggested questions do have some hints, but I do not see how to apply their answers to my question. E.g., the suggestion to use default value for the argument at https://stackoverflow.com/a/19837683/15270325 does not seems to work here. – Max Alekseyev Feb 23 '21 at 22:19
  • The late binding explanations in the Creating Functions in a loop question applies here - it is essentially what Alain T. is saying in the answer you accepted. – wwii Feb 23 '21 at 22:43

1 Answers1

2

The two lambdas in f_new actually call the same function.

This is because the list is formed of lambdas that capture the fi variable (at compile time) but do not actually execute the function. So, when the list runs through the generator ... for fi in f the lambdas that are produce all use the captured variable fi and end up with the same function pointer (i.e. the last one in f)

You would need to consume the current value of fi in the comprehension in order to avoid this capture side effect:

f_new = [(lambda fn:lambda x: fn(x))(fi) for fi in f]
print( [fi(2) for fi in f_new] )
[4, 102]
Alain T.
  • 40,517
  • 4
  • 31
  • 51
  • This explains it! Thank you! – Max Alekseyev Feb 23 '21 at 22:23
  • A follow-up question: from this perspective, what is the subtle difference between `chain()` and `chain.from_iterable()` producing different results in my original code example? Just in case, the code is [available at Sagecell](https://sagecell.sagemath.org/?z=eJy1jsFOwzAQRO_5ilFPdoiqEm6RcuTAB3BCqHLdNazq2JGzEf58FonSHjiABJeVZkaz83iacxGwUJGc49I0ASNMdNPh6FAH1LbtO1x039ZrWW9udzvbNHPhJGZzX2fyQkcUWtYow8Z-Joi86P3a2fpXx2kbSp72H6Y7RLqOnffrtEYnZIpLL2TubIewJj-et7v9gMCmqs-JhV0ce4uQi7rqIMBanOcvhI-Jfsdo0Jo_5bLfgz0IYs6nRSlOBA54I21U5dTeiPDUP_8Lqv79Mew7zCy3Ng==&lang=sage&interacts=eJyLjgUAARUAuQ==). – Max Alekseyev Feb 23 '21 at 23:30
  • 1
    from_iterable will finish consuming the first result of `for fi in f` before going to the next value of `fi`. This will effectively avoid the captureside effect because the function in the current `fi` is applied to the range() before `fi` gets its next value. chain() evaluates all the parameters beforehand so all lambdas are generated before it starts evaluating the ranges which falls in the capture trap. – Alain T. Feb 23 '21 at 23:42
  • Thank you! Returning to the current code in question, it seems that the easiest fix is to create `f_new` as a generator rather than a list: `f_new = (lambda x: fi(x) for fi in f)` – Max Alekseyev Feb 23 '21 at 23:50
  • For the direct execution of the function, it would indeed be enough. If you're going to use the function in other iterators, you'll need to be careful to ensure lazy evaluation throughout. – Alain T. Feb 23 '21 at 23:52
  • Can you please illustrate a possible side effect of this kind with use in other iterators? – Max Alekseyev Feb 23 '21 at 23:55
  • For example `[ fi(i) for i in range(3) for fi in f_new ]` would not work because the f_new generator is consumed on the first iteration of range(3) forcing you to make it a list or something which would cause the lambdas to be created ahead of time (ending up all on the last function of `f`). It's not impossible to work around but you have to keep that in mind every time. – Alain T. Feb 24 '21 at 00:06
  • Hmm. With `f_new = (lambda x: fi(x) for fi in f)`, your code `[ fi(i) for i in range(3) for fi in f_new ]` on the first run produces `[0, 100]` and if repeated it gives `[]`, which is totally puzzling. Illustration: https://sagecell.sagemath.org/?q=bpcvto – Max Alekseyev Feb 24 '21 at 00:17
  • It is what I predicted. The `for fi in f` loop is called for every i in range(3) but after going through functions for i=0, the f_new generated won't return any more values for i=1 and i=2 because it already reached the end. – Alain T. Feb 24 '21 at 00:24
  • I see. So, it's essentially a one-time-use generator. – Max Alekseyev Feb 24 '21 at 03:13