python: dict of (lambda) functions

Question

I've experienced some strange behaviour when storing lambda functions into a dictionary: If you try to pass some default value to a function in a loop, only the last default value is being used.

Here some minimal example:

#!/usr/bin/env python
# coding: utf-8

def myfct(one_value, another_value):
    "do something with two int values"
    return one_value + another_value

fct_dict = {'add_{}'.format(number): (lambda x: myfct(x, number))
            for number in range(10)}

print('add_3(1): {}, id={}'.format(fct_dict['add_3'](1), id(fct_dict['add_3'])))
print('add_5(1): {}, id={}'.format(fct_dict['add_5'](1), id(fct_dict['add_5'])))
print('add_9(1): {}, id={}'.format(fct_dict['add_9'](1), id(fct_dict['add_9'])))

The output reads as follows

add_3(1): 10, id=140421083875280
add_5(1): 10, id=140421083875520
add_9(1): 10, id=140421083876000

You get dissimilar functions (id not identical) but every function uses the same second argument.

Can somebody explain what's going on?

The same holds with python2, python3, pypy...

are you trying to use 'list comprehension' inside a dictionnary declaration? — tglaria, Jan 18 '16 at 12:09
correct: I use it sometimes to create a bunch of entries/functions if I need distinct names -> see https://www.python.org/dev/peps/pep-0274/ for dict comprehension — Hensing, Jan 18 '16 at 12:27

Dirk · Accepted Answer · 2016-01-18T12:21:36.067

The fix:

def make_closure(number):
    return lambda x: myfct(x, number)

used as

{'add_{}'.format(number): make_closure(number) for number in range(10)}

The reason for this behaviour is, that the variable number (think: named memory location here) is the same during all iterations of the loop (though its actual value changes in each iteration). "Loop" here refers to the dictionary comprehension, which internally is based on a loop. All lambda instances created in the loop will close over the same "location", which retains the value last assigned to it (in the last iteration of the loop).

The following code is not what actually happens underneath. It is merely provided to shed light on the concepts:

# Think of a closure variable (like number) as being an instance
# of the following class

class Cell:
    def __init__(self, init=None):
        self.value = None

# Pretend, the compiler "desugars" the dictionary comprehension into
# something like this:

hidden_result_dict = {}
hidden_cell_number = Cell()

for number in range(10):
    hidden_cell_number.value = number
    hidden_result_dictionary['add_{}'.format(number)] = create_lambda_closure(hidden_cell_number)

All lambda closures created by the create_lambda_closure operation share the very same Cell instance and will grab the value attribute at run-time (i.e., when the closure is actually called). By that time, value will refer to last value ever assigned to it.

The value of hidden_result_dict is then answered as the result of the dict comprehension. (Again: this is only meant as be read on a "conceptual" level; it has no relation to the actual code executed by the Python VM).

I found your answer much better than those to the question this is a duplicate of. You might want to add your answer there as well, so more people see it. — Christian Geier, Jan 18 '16 at 20:57
Thanks so much, I was going crazy because of this error and wasted 3 hours on trying to understand what is happening — Guiste, Nov 14 '18 at 06:21

score 3 · Answer 2 · answered Jan 18 '16 at 12:13

number is a variable which has different value for each iteration of the dict comprehension. But when you do lambda x: myfct(x, number), it does not use value of number. It just creates a lambda method that will use value of number when it will be called/used. So when you use you add_{} methods, number has value 9 which is used in every call to myfct(x, number).

python: dict of (lambda) functions

2 Answers2

Linked

Related