Issue with Python3's built-in zip function

Question

Python 3.4.2 (default, Oct  8 2014, 13:44:52) 
[GCC 4.9.1 20140903 (prerelease)] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> gen = (x for x in range(10)) ## Need to wrap range into ()'s to create a generator, next(range(10)) is invalid
>>> list(zip(gen, [1,2,3])) ## zip will "eat up" the number 3
[(0, 1), (1, 2), (2, 3)]
>>> next(gen) ## Here i need next to return 3
4
>>>

The problem is that I'm losing a value after the zip call. This would be a bigger issue had it not been for the fact that gen is pure code.

I don't know whether or not it would be possible to create a function that behaves like this, it's definitely possible if only one of the arguments to the zip function is a generator and the rest are "normal" iterators where all the values are known, and stored in memory. If that were the case you could just check the generator last.

Basically what I am wondering is if there is any function in the python standard library that will act like I need it to in this case.

Of course, in some cases one could just do something like

xs = list(gen)

Then you only have to deal with a list.

I could also add, that getting the last value that zip got from gen would also be a solution to this problem.

Rather than produce a generator expression, just use `iter(range(10))`. — Martijn Pieters, Oct 23 '14 at 16:42
possible duplicate of [Why does zip() drop the values of my generator?](http://stackoverflow.com/questions/11210300/why-does-zip-drop-the-values-of-my-generator) — Stephane Rolland, Oct 23 '14 at 16:44
@StephaneRolland not a duplicate, since that question asks why it does that and this one asks how to work around it. — Mark Ransom, Oct 23 '14 at 16:47

Martijn Pieters · Accepted Answer · 2014-10-23T16:55:20.283

No, there are no built-in functions that avoid this behaviour.

What happens is that the zip() function tries to get the next value for all inputs, so that it can produce the next tuple. It has to do this in a order, and it is only logical that that order is the same as the arguments passed in. In fact, the order is guaranteed by the documentation:

The left-to-right evaluation order of the iterables is guaranteed

Because the function needs to support arbitrary iterables, zip() does not make any attempts at determining the length of all parameters. It doesn't know that your second parameter only has 3 elements. It simply tries to get the next value for each of the parameters, builds a tuple and returns that. If any of the parameters cannot produce a next value, the zip() iterator is done. But that does mean that it'll ask your generator for the next element first before asking the list.

Apart from altering the order of your inputs, you can build your own zip() function instead, that does try to take length into account, where available:

def limited_zip(*iterables):
    minlength = float('inf')
    for it in iterables:
        try:
            if len(it) < minlength:
                minlength = len(it)
        except TypeError:
            pass
    iterators = [iter(it) for it in iterables]
    count = 0
    while iterators and count < minlength:
        yield tuple(map(next, iterators))
        count += 1

So this version of the zip() function tries to get a bead on the minimal length of any sequences you passed in. This does not protect you from using a shorter iterable in the mix, but does work for your test case:

Demo:

>>> gen = iter(range(10))
>>> list(limited_zip(gen, [1, 2, 3]))
[(0, 1), (1, 2), (2, 3)]
>>> next(gen)
3

Giving the generator as the second argument to zip seems to work for my case. Since it will go from left to right in the argument list. — A_User, Oct 23 '14 at 16:55
@Yoanar: sure, but I couldn't assume that changing the order is a workable solution for everyone. — Martijn Pieters, Oct 23 '14 at 16:57

Irshad Bhat · Answer 2 · 2014-10-23T17:04:34.660

2

Problem is that zip(gen,[1,2,3]) generates 0,1,2, and 3 also but finds that second argument is of length three only. So if you do it in reverse, you can generate 3 in next(gen) code line:

>>> gen = (x for x in range(10))
>>> list(zip([1,2,3],gen))
[(1, 0), (2, 1), (3, 2)]
>>> next(gen)
3

edited Oct 23 '14 at 17:04

answered Oct 23 '14 at 16:54

Irshad Bhat

8,479
1
26
36

score 1 · Answer 3 · answered Oct 23 '14 at 16:54

The problem is that when zip reaches StopIteration on one of its iterables, it forgets the values returned from the preceding iterables.

Here's a solution using zip_longest and groupby from itertools to partition the zip sequence into before and after the shortest iterable terminates:

>>> from itertools import zip_longest, groupby
>>> sentinel = object()
>>> gen = (x for x in range(10))
>>> g = iter(groupby(zip_longest(gen, [1,2,3], fillvalue=sentinel),
...                  lambda t: sentinel not in t))
>>> _, before = next(g)
>>> list(before)
[(0, 1), (1, 2), (2, 3)]
>>> _, after = next(g)
>>> next(after)
(3, <object object at 0x7fad64cbf080>)
>>> next(gen)
4

Mark Ransom · Answer 4 · 2014-10-23T17:04:47.817

You can use a wrapper class around your generator to give you access to the most recent element. I took most of this code from the Python Wiki at https://wiki.python.org/moin/Generators.

class gen_wrap(object):
    def __init__(self, gen):
        self.gen = gen
        self.current = None

    def __iter__(self):
        return self

    # Python 3 compatibility
    def __next__(self):
        return self.next()

    def next(self):
        self.current = next(self.gen)
        return self.current

    def last(self):
        return self.current

>>> gen = gen_wrap(x for x in range(10))
>>> list(zip(gen, [1,2,3]))
[(0, 1), (1, 2), (2, 3)]
>>> gen.last()
3

Issue with Python3's built-in zip function

4 Answers4