generating functions in a loop using variables from the loop cause shadowing effects of variables

Question

I want to generate small functions in a loop that access variables from the loop. Then I want to compose and apply the functions all at once. An MWE looks lie this:

from functools import reduce

def compose(*funcs):
    return lambda x: reduce(lambda y, f: f(y), reversed(funcs), x)


def replace_values_by_type(text, values, types) -> str:

    def peek(x):
        print('peek:', x)
        return x

    replacements = [lambda txt: peek(peek(txt).replace(peek(val), f'<{peek(typ)}>')) for val, typ in zip(values, types)]
    replace = compose(*replacements)

    text_with_replacements = replace(text)

    print(values)
    print(types)
    print(text)
    print(text_with_replacements)
    print()

    return text_with_replacements



replace_values_by_type('this is a test sentence', ['is', 'test'], ['A', 'B'])

When I run this I expected to get "this <A> a <B> sentence". But Only the last pair of val and typ from the loop actually are used. So I guess some shadowing or overwriting happens. Can you explain this?

-> % py mwe.py 
peek: this is a test sentence
peek: test
peek: B
peek: this is a <B> sentence
peek: this is a <B> sentence
peek: test
peek: B
peek: this is a <B> sentence
['is', 'test']
['A', 'B']
this is a test sentence
this is a <B> sentence

Btw. to isolate the issue I also wrote the function like this:

def replace_values_by_type(text, values, types) -> str:

    replacements = []

    for val, typ in zip(values, types):
        def f(txt):
            return txt.replace(val, f'<{typ}>')

        replacements.append(f)

    text_with_replacements = text
    for f in replacements:
        text_with_replacements = f(text_with_replacements)

    return text_with_replacements


print(replace_values_by_type('this is a test sentence', ['is', 'test'], ['A', 'B']))

The problem remains the same.

I think the list comprehension just establishes a single scope for the iteration variables, there isn't a scope for each iteration. — Barmar, Oct 13 '20 at 21:44
@Bramar So they get overwritten each iteration then, huh? I tried to use deepcopy on `val` and `typ` inside `f`, because I suspected that. But that did not help either. — lo tolmencre, Oct 13 '20 at 21:46
There's nothing to copy. The issue is the uses of the variables `val` and `typ` -- all the lambdas refer to the same variables, and they contain the last values from the loop. — Barmar, Oct 13 '20 at 21:48
In JavaScript this is called the "infamous loop issue". I'm trying to find similar questions about Python. — Barmar, Oct 13 '20 at 21:49
Similar: https://stackoverflow.com/questions/54288926/python-loops-and-closures — Barmar, Oct 13 '20 at 21:50
Is there any elegant solution to create the `f` closures that fix the values of `val` and `typ` as they were when `f` was created? — lo tolmencre, Oct 13 '20 at 21:55

score 2 · Accepted Answer · edited Oct 13 '20 at 22:05

All the closures created by the list comprehension are in the same variable scope, and there's just a single instance of the val and typ variables for the loop. When the closures are called later, the variables have their values from the last iteration.

You need to generate the closures in a unique scope for each iteration. One way to do this is to write a separate function that returns the closures, since every function establishes a new scope.

def replacer(val, typ):
    return lambda txt: peek(peek(txt).replace(peek(val), f'<{peek(typ)}>'))

replacements = [replacer(val, typ) for val, typ in zip(values, types)]

Ah, I understand, thank you very much. – lo tolmencre Oct 13 '20 at 22:07 — lo tolmencre, Oct 13 '20 at 22:07

generating functions in a loop using variables from the loop cause shadowing effects of variables

1 Answers1