How to use with statement in lambda expression for Python

Question

I am using Python 3.5.2 (the one used for MS SQL Server Machine Learning Services). I would like to have a lambda expression that uses with so not to repeat an expensive calculation in a tuple. A minimal example is

lambda x : 
  with x**2 as x2, x**3 as x3 :
    [ (x2**3)*x3-x3 , x2, (x3-5)**x2 ]

However, this returns a syntax error. How is a with statement used in a lambda expression?

I am coming from Wolfram Language where this can be done. For example

With[{x2 = #^2, x3 = #^3},
  { (x2^3)*x3-x3 , x2, (x3-5)^x2 }
]&

I have a collection of these tiny calculations that are only needed in a small part of the code (within another function). I want to hold these as lambda functions to select and apply/map based particular conditions in that function.

Pretty sure this isn't possible; it's just not what lambdas are designed for. Make a function. — glibdud, Sep 17 '19 at 14:23
A Lambda can contain exactly one *expression*, so this isn't possible by definition. — deceze, Sep 17 '19 at 14:24
@khelwood I have a collection of tiny calculations that are only needed in a small part of the code (within another function). So it seems reasonable to hold these as lambda functions to select based particular conditions in that function. — Edmund, Sep 17 '19 at 14:32
@Edmund as "reasonnable" as it might seem, that's not how python works (neither `lambda` nor `with`). The fact that it can be done that way in another language is also irrelevant - by definition, different languages are, well, different, you know ?-) — bruno desthuilliers, Sep 17 '19 at 14:39
@brunodesthuilliers Not saying the fact that it can be done that way in another language means it should be able to be done that way in Python. Merely providing background as to why I thought this was possible and may have been missing the correct syntax to do so. No need to be so salty, mate. — Edmund, Sep 17 '19 at 14:42
@Edmund sorry, but it just looked like you expected Python to work just like Wolfram. This being said, you'll probably save yourself a lot of time, pain and frustration by doing the official Python tutorial (and then browsing the doc). Also, note that while Python has some support for functional programming idioms, it's still not a proper functional language - it's really an object language -, so functional idioms might not always be the most efficients. — bruno desthuilliers, Sep 17 '19 at 14:50
Oh and yes: if you have a lot of numeric calculations to do, you may want to have a look at packages like numpy. — bruno desthuilliers, Sep 17 '19 at 14:51
@brunodesthuilliers Yes, not as near to a functional programming as I initially thought. Now that I am using it it is definitely OOP with a slight wink to functional programming. — Edmund, Sep 17 '19 at 18:50

chepner · Answer 1 · 2019-09-17T15:00:25.630

3

In Python 3.8, you can write

lambda x: [ (x2 := x**2) * (x3:=x**3) - x3, x2, (x3 - 5) ** x2 ]

However, that's not terribly readable. I would still just use a def statement to define a named function, as shown in https://stackoverflow.com/a/57976808/1126841.

(Perhaps some future version of Python will find a way to support functional-style let expressions like

let x2 := x**2, x3 := x**3 in (x2*x3 - x3, x2, (x3-5) ** x2)

but I wouldn't hold my breath. This really requires the keywords for readability, and there is strong resistance--with good reason, I'll admit--to introducing new keywords or overloading existing keywords.)

edited Sep 17 '19 at 15:00

answered Sep 17 '19 at 14:55

chepner

497,756
71
530
681

Thanks for the insight. If SQL Server upgrades their Machine Learning Services to 3.*+ then I might switch to this implementation. (+1) – Edmund Sep 17 '19 at 18:43

bruno desthuilliers · Accepted Answer · 2019-09-18T07:59:38.367

In python, this is spelled:

def fun1(x):
    x2 = x**2
    x3 = x**3 
    return [(x2**3) * x3 - x3 , x2, (x3 - 5) ** x2]

EDIT:

A couple more points...

First: someone mentions in a comment tha you could cache computation results for improved performances. It's indeed a common pattern usually named memoization, which is rather easy to implement in Python. Just note that this only works properly with pure functions, and is of course only beneficial if the memoized function is repeatedly called with the same arguments.

Second point, you mention that:

I have taken this route and defined the mini functions within the function.

You have to know that in Python, def is an executable statement that creates a new function object (yes, Python functions are objects - actually everything in Python is an object) when executed.

This means that "top-level" functions (functions defined at the top-level of a module) are created once when the module is first imported in the current process, but that inner (nested) functions are created anew each time the outer (enclosing) function is called.

Also, Python functions are closures, so a nested function can capture the environment of the outer function (and it's the main - and only as far as I'm concerned - reason for using nested functions in Python).

This means that you can use inner functions as another way to cache some precomputed values and make them available to a set of functions, ie:

def outer(x, y):
    x2 = x ** 2
    x3 = x ** 3

    def func1():
        return [(x2**3) * x3 - x3 , x2, (x3 - 5) ** x2]

    def func2(z):
        # dummy example
        return (x2**y) + (y - x3 ** z)

    # etc

    results = [func2(i) for i in func1()]
    return results

On the other hand, if your "helper" functions are not dependant on their enclosing scope, the only thing you get from defining them within the outer function is the overhead of creating the same functions again and again and again each time the outer function is called. It's of course not a problem if the outer func is only called once or twice per process, but if it's invoked in a tight loop the overhead can quickly become an issue.

So the only way to proceed is to create named functions for all of these small calculations and then reference them by name in the dataframe column? — Edmund, Sep 17 '19 at 14:53
@Edmund Not by name. You pass the function object. Where you're currently doing `lambda: ...`, you instead do `fun1`. And yes, they will have to be named. — deceze, Sep 17 '19 at 14:58
If you want to avoid repeating your expensive calculation, it only makes sense you cache the results somehow i.e. naming them to be referred back later. Even in your example your logic is essentially naming them, just in a different context that's not valid syntax for Python. — r.ook, Sep 17 '19 at 15:00
I have taken this route and defined the mini functions within the function. If the number of them grows I will most likely pull them out into a package. But this is working for now. Thanks! — Edmund, Sep 17 '19 at 18:45
@Edmund beware that in Python `def` is an executable statement, so inner (nested) function are created anew on call of the outer function. Depending on your use case this may (or not) have some impact on performances. — bruno desthuilliers, Sep 18 '19 at 06:46
@Edmund you may want to check my edited answer for more on inner functions and how to use (or not to use) them ;-) — bruno desthuilliers, Sep 18 '19 at 08:00

How to use with statement in lambda expression for Python

2 Answers2