python lambda functions defined in a loop, end up pointing to the same operation

Question

I was looking for an algorithm capable of storing and running functions with a flexible number of arguments. I ended up finding a switch/case analogue for python which meets my requirements:

def opA_2(x, y):
    return x - y


def opB_3(x, y, z):
    return x + y - z


def opC_2(x, y):
    return x * y


def opD_3(x, y, z):
    return x * y + z


op_dict = {'opA_2': opA_2,
           'opB_3': opB_3,
           'opC_2': opC_2,
           'opD_3': opD_3
           }

op_lambda_dict = {'opA_2': lambda x, y, kwargs: op_dict['opA_2'](x, y),
                  'opB_3': lambda x, y, kwargs: op_dict['opB_3'](x, y, kwargs['z']),
                  'opC_2': lambda x, y, kwargs: op_dict['opC_2'](x, y),
                  'opD_3': lambda x, y, kwargs: op_dict['opD_3'](x, y, kwargs['z']),
                  }


def dispatch_op(func_dict, op, x, y, **kwargs):
    return func_dict.get(op, lambda a, b, c: None)(x, y, kwargs)

coefs_dict = {'i': 1, 'j': 2, 'k': 3, 'z': 4}

print('Original lambda dict result:', dispatch_op(op_lambda_dict, 'opB_3', 1, 2, **coefs_dict))

Resulting in:

Original lambda dict result: -1

Once I implemented this structure to my target code, however, I encountered many issues because my operations are defined via a loop.

As far as I understand it, this is because the lambda functions are not initialised, and they end up pointing to the last operation declared.

This additional code reproduces the issue:

op_looplambda_dict = {}
for label, func in op_dict.items():
    if '2' in label:
        op_looplambda_dict[label] = lambda x, y, kwargs: func(x, y)
    if '3' in label:
        op_looplambda_dict[label] = lambda x, y, kwargs: func(x, y, kwargs['z'])

print('Loop lambda dict result:', dispatch_op(op_looplambda_dict, 'opB_3', 1, 2, **coefs_dict))

Resulting in:

Loop lambda dict result: 6

This is the result of opD_3 instead of opB_3

I wonder if anyone could please offer any advice, on how to properly declare the lambda functions in the second case or a different code structure to avoid it. Thanks a lot.

Answer 1

The problem is to do with scope: each lambda function creates a "closure" over the func variable, and can access that variable's current value when the function runs. And because it's not running until the loop is complete, the value it uses is the last one it held in the loop.

The solution is therefore to utilise scope to your advantage, by ensuring that each lambda closes over a func variable which only ever holds one value - the value you need. You need a new scope for each iteration of the loop. And the only way Python allows you to create scope is with a function.

So the solution will look something like this:

op_looplambda_dict = {}
for label, func in op_dict.items():
    def add_to_dict(function):
        if '2' in label:
            op_looplambda_dict[label] = lambda x, y, kwargs: function(x, y)
        if '3' in label:
            op_looplambda_dict[label] = lambda x, y, kwargs: function(x, y, kwargs['z'])
    add_to_dict(func)

I'd be the first to admit that this is a little clumsy, but it should work. The only difference from your code is that I've put the loop body inside a function - then immediately called that function each time. So there's no difference in how it behaves, except for the how the variables are scoped - which is key here. When those lambda function run, they will use the value of function from their immediately enclosing scope, and because off the add_to_dict function having its own scope, that will be a different function for each time through the loop.

I would encourage you to look at this Q&A for useful background - it's about Javascript rather than Python, but the scoping mechanisms (at least in "old-school" Javascript before ES6) are identical between the two languages, and the underlying issue you have here is identical to the one in this much-asked JS question. Unfortunately many of the solutions that are available in modern Javascript aren't applicable to Python, but this one does - a simple adaptation of the "Immediately Invoked Function Expression" (IIFE for short) pattern that's common in JS.

And as you can probably tell, I'm personally more experienced with JS than with Python, so I'd be interested to hear if there's a nicer, more idiomatic, Python solution to this problem than what I've done above.

Answer 2

The functions can be added in the dictionary using their names.

When you dispatch a call to an operation, you could compare the received arguments with signature of matched function in the registry. This way you could figure out other arguments for that function from the optional arguments passed in the dispatch call. For example,

import inspect
import functools

registry = {}

def register_operation(func):
    if func.__name__ in registry:
            raise TypeError("duplicate registration")
    func.__signature = inspect.signature(func)
    registry[func.__name__] = func
    return func

def dispatch_operation(func_name, *args, **kwargs):
    func = registry.get(func_name, None)
    if not func:
        raise TypeError("no match")
    rest_parameters = list(func.__signature.parameters)[len(args):]
    rest_args = (kwargs.get(p) for p in rest_parameters)

    ba = func.__signature.bind(*args, *rest_args)
    ba.apply_defaults()

    return func(*ba.args, **ba.kwargs)

@register_operation
def opA_2(x, y):
    return x - y

@register_operation
def opB_3(x, y, z):
    return x + y - z

@register_operation
def opC_2(x, y):
    return x * y

@register_operation
def opD_3(x, y, z):
    return x * y + z


coefs_dict = {'i': 1, 'j': 2, 'k': 3, 'z': 4}

print('Original dict result:', dispatch_operation('opB_3', 1, 2, **coefs_dict))