itertools.product function creation producing unexpected results

Question

I'm having a little trouble understanding the results of the below snippet, and think it's because I'm confused about function binding. Why would the following snippets produce different results?

import itertools

def make_funcs(lst):
    for val in lst:
        def f():
            return sum(1 for i in range(10) if i > val)
        f.func_name = ">" + str(val)
        yield f
## examples:
for f in make_funcs(range(2)):
    print(f.func_name, f())
## prints:
>0 9
>1 8

## works as expected:
for f in make_funcs(range(2)):
    for g in make_funcs(range(2)):
        print(f.func_name, g.func_name, f() + g())

## prints:
>0 >0 18
>0 >1 17
>1 >0 17
>1 >1 16

while on the other hand:

## provides results that are counter-intuitive (to me, at least)
for f, g in itertools.product(make_funcs(range(2)), make_funcs(range(2))):
    print(f.func_name, g.func_name, f() + g())

## prints:
>0 >0 16
>0 >1 16
>1 >0 16
>1 >1 16

It appears to me that it's only grabbing/using/binding the last variable in each implicit for-loop for the calculation, even though it's grabbing the correct variable for the function names.

What am I missing about scoping or function definitions or closures (or whatever) that's causing these results?

NB: if any of the tags I put on this question are irrelevant, feel free to remove them - I put them all because I'm not sure what the issue is.

Related: [Local variables in Python nested functions](http://stackoverflow.com/q/12423614/190597). — unutbu, Aug 03 '15 at 00:50
@unutbu more than related - it was the same thing. Thanks for the link, that coupled with one of the answers below clears it all up pretty well! — dwanderson, Aug 04 '15 at 22:27

score 2 · Answer 1 · answered Aug 05 '15 at 07:47

The other answer explains why you see what you see -- it's because functions capture outside variables by reference, and the same val variable is captured by the several functions, who see the changes to the variable.

To add, if you want to avoid this, if you want to capture by value, one way to do it is to use a parameter and default argument in the inner function:

def make_funcs(lst):
    for val in lst:
        def f(val=val):
            return sum(1 for i in range(10) if i > val)
        f.func_name = ">" + str(val)
        yield f

Martin · Accepted Answer · 2015-08-11T08:32:40.413

All functions still reference the variable val.

def make_funcs(lst):
    a = []
    for val in lst:
        def f():
            return sum(1 for i in range(10) if i > val)
        f.func_name = ">" + str(val)
        a.append(f)
    return a

Results in all prints being "counter intuitive".

def make_funcs(lst):
    a = []
    for val in lst:
        def f():
            return sum(1 for i in range(10) if i > val)
        f.func_name = ">" + str(val)
        a.append(f)
    val = 10
    return a

Always results in 0's.

However, because you use a generator, the value of val is only changed after it has been used, so all seems well. When using itertools.product, the docs says it does this:

def product(*args, repeat=1):
    # product('ABCD', 'xy') --> Ax Ay Bx By Cx Cy Dx Dy
    # product(range(2), repeat=3) --> 000 001 010 011 100 101 110 111
    pools = [tuple(pool) for pool in args] * repeat
    result = [[]]
    for pool in pools:
        result = [x+[y] for x in result for y in pool]
    for prod in result:
        yield tuple(prod)

Which means it first iterates over both generators (effectively changing the value of val four times) and only then calculates the results.

This all happens because val is defined in the scope of make_funcs (and not in the scope of f), so if a second call to the generator changes the value of val, all functions reference the new value.

Edit: Please also read the answer by @newacct

itertools.product function creation producing unexpected results

2 Answers2