Understanding buffering of tee

Question

After reading this answer which involves tee in its code, I played a little bit, trying to generalize the solution presented there.

The task was to lazily "transpose" a list. That is, the list [xrange(1, 4), xrange(4, 7), xrange(7, 10)] (in python 2.7) would have to become something which is similar to [xrange(1, 10, 3), xrange(2, 10, 3), xrange(3, 10, 3)].

I copy here the solution presented there:

def transpose(iterable_of_three_tuples):
    teed = itertools.tee(iterable_of_three_tuples, 3)
    return (e[0] for e in teed[0]), (e[1] for e in teed[1]), (e[2] for e in teed[2])

My generalization was about trying to do the same, where the 3 is a parameter. Since 3 is the length of the input generators, you can't guess it, and you have to add it as an argument for the function. Since some generators may be large or even infinite, I also think of this parameter as a limit, so we take the first n elements from each input generator only.

This is my code:

def transpose(zipped, n):
    teed = tee(zipped, n)
    return [(e[i] for e in teed[i]) for i in xrange(n)]

This is not working as expected, as the following example shows:

>>> data = [xrange(1, 4), xrange(4, 7), xrange(7, 10)]
>>> x, y, z = transpose(data, 3)
>>> x.next()
3

(I expected 1).

I guess this is something about the generator inside a list comprehension and leak of names, or lazy evaluation of i or something similar... Can you tell what's wrong with my code and how it can be fixed?

Your issue is similar to that discussed in [this question](http://stackoverflow.com/questions/13355233/python-lambda-closure-scoping). That question talks about lambdas/functions, but the scoping rules for generator comprehensions are the same. See if that question helps you understand what's going on. — BrenBarn, Apr 17 '14 at 07:18
I am actually familiar with that scoping issues with functions; setting a default argument (`n=n`) is a workaround which shifts the name into the inner scope. However, I'll read you solution there carefully and try to apply it on my own problem. — Bach, Apr 17 '14 at 07:22
You said it right: the generator closure closes over names, not values. — Bach, Apr 17 '14 at 07:24

score 0 · Accepted Answer · answered Apr 17 '14 at 07:32

Thanks to @BrenBarn's comment, here's a (quite ugly) patchy fix. You are welcome to propose better ones, though.

def transpose(zipped, n):                          
    teed = tee(zipped, n)                            
    for i in xrange(n):
        gen = lambda teed, i=i: (e[i] for e in teed[i])
        yield gen(teed)

Understanding buffering of tee

1 Answers1