Python doit: subtask yields same uptodate callable

Question

I have this code in dodo.py:

def closed_over(par):
    print("\n??? " + par)
    if par == "bar":
        return False
    else:
        return True

def task_bug():
    for par in ("foo", "bar"):
        print("par: " + par)
        # closure!
        exist_fn = lambda: closed_over(par)
        print(exist_fn)

        yield {
            "name": par,
            "actions": [["echo", "action:", par]],
            "verbosity": 2,
            "uptodate": [exist_fn]
        }

When I run doit bug:foo I expect to NOT execute it as (closed_over returns True), but:

par: foo
<function task_bug.<locals>.<lambda> at 0x7f8926f9a560>
par: bar
<function task_bug.<locals>.<lambda> at 0x7f8926f9a5f0>

??? bar      <- par should be foo
.  bug:foo
action: foo  <- echo was called

As you can see above the two closures outside the yield are different function objects but for some reason uptodate is always calling the same.

Does this answer your question? [Creating functions in a loop](https://stackoverflow.com/questions/3431676/creating-functions-in-a-loop) — Karl Knechtel, Aug 18 '22 at 02:09

score 2 · Answer 1 · answered Dec 01 '19 at 08:12

2

Your problem is regarding python closure scoping (it scope variables not values)... Nothing to do with doit. Probably have many questions about this here on Stack Overflow :)

Python lambda closure scoping

As you can see above the two closures outside the yield are different function objects but for some reason uptodate is always calling the same.

No, it is not calling same closure. The problem is the same variable as parameter.

answered Dec 01 '19 at 08:12

schettino72

2,990
1
28
27

thanks you, I was worried I did not understand something properly – Andrea Richiardi Dec 02 '19 at 18:08

Andrea Richiardi · Answer 2 · 2022-08-19T19:21:07.793

0

As it turns out, the problem seems related to python's closure late binding.

According to a couple of answers in the wild, this is what actually works:

from functools import partial

def closed_over(par):
    print("\n??? " + par)
    if par == "bar":
        return False
    else:
        return True

def task_bug():
    """Test a doit bug."""

    for par in ("foo", "bar"):
        print("par: " + par)
        exist_fn = partial(closed_over, par)
        print(exist_fn)

        yield {
            "name": par,
            "actions": [["echo", "action:", par]],
            "verbosity": 2,
            "uptodate": [exist_fn]
        }

$ doit bug
par: foo
functools.partial(<function closed_over at 0x7f6ab3eb6cb0>, 'foo')
par: bar
functools.partial(<function closed_over at 0x7f6ab3eb6cb0>, 'bar')

??? foo
-- bug:foo

??? bar
.  bug:bar
action: bar

Thanks Karl for correcting my initial guess.

edited Aug 19 '22 at 19:21

answered Dec 02 '19 at 19:38

Andrea Richiardi

703
6
21

This isn't a dynamic vs lexical scoping issue. There are other lexically scoped languages, such as SML (as I learned while closing other duplicate questions), where it would work the way you want. Instead, the issue is that Python is *late binding* for closures: the name is looked up when it is used, not when the closure is created. (A combination of late binding and dynamic scoping would make the problem even worse: the closure could pick up an *entirely unrelated* value for `par` from the scope *where it is called*). – Karl Knechtel Aug 18 '22 at 02:12
Thank you this makes more sense indeed, I will update the answer – Andrea Richiardi Aug 19 '22 at 19:19

Python doit: subtask yields same uptodate callable

2 Answers2