zip-like function that fails if a particular iterator is not consumed

Question

I would like a zip like function that fails if the right-most iterator is not consumed. It should yield until the failure.

For example

>>> a = ['a', 'b', 'c']
>>> b = [1, 2, 3, 4]

>>> myzip(a, b)
Traceback (most recent call last):
    ...
ValueError: rightmost iterable was not consumed

>>> list(myzip(b, a))
[(1, 'a'), (2, 'b'), (3, 'c')]

Perhaps there a function in the standard library that can help with this?

Important Note:

In the real context the iterators are not over objects so I can't just check the length or index them.

Edit:

This is what I have come up with so far

def myzip(*iterables):
    iters = [iter(i) for i in iterables]

    zipped = zip(*iters)

    try:
        next(iters[-1])
        raise ValueError('rightmost iterable was not consumed')
    except StopIteration:
        return zipped

Is this the best solution? It doesn't keep the state of the iterator because I call next on it, which might be a problem.

Note that the code you edited in for `myzip()` won't work - it'll always give you the `ValueError` as long as the last iterable has at least one element in it to begin with. This is because - unlike in Python 2, where `zip()` will consume the iterables immediately, and return a list - in Python 3 `zip()` returns a generator and consumes the iterables lazily. This means that when you call `next()` in the current version of `myzip()`, no items have been consumed by `zip()` yet, and all the iterators are still at the very beginning. — Aleksi Torhamo, Nov 11 '18 at 13:47
Yeah I did realize this when I tried to use this, I am using a solution that uses `yield from` like in your answer. — Ross MacArthur, Nov 12 '18 at 06:54

Aleksi Torhamo · Accepted Answer · 2018-11-11T09:19:29.497

1

There's a few different ways you can go about doing this.

You could use the normal zip() with an iterator and manually check that it gets exhausted.

def check_consumed(it):
    try:
        next(it)
    except StopIteration:
        pass
    else:
        raise ValueError('rightmost iterable was not consumed')

b_it = iter(b)
list(zip(a, b_it))
check_consumed(b_it)

You could wrap the normal zip() to do the check for you.

def myzip(a, b):
    b_it = iter(b)
    yield from zip(a, b_it)
    # Or, if you're on a Python version that doesn't have yield from:
    #for item in zip(a, b_it):
    #    yield item
    check_consumed(b_it)

list(myzip(a, b))

You could write your own zip() from scratch, using iter() and next().

(No code for this one, as option 2 is superior to this one in every way)

edited Nov 11 '18 at 09:19

answered Nov 11 '18 at 09:16

Aleksi Torhamo

6,452
2
34
44

Is there a way to peek the iterator, so that its state is not modified. – Ross MacArthur Nov 11 '18 at 09:19
@Ross: If you mean "get an error when `a` has 3 items and `b` has 4 items, without consuming any items from either", the answer is "no", AFAIK. Even if you just don't want to consume one extra item from `b`, I think the answer is still "no". You *could* use `itertools.chain()` to "add the consumed items back afterwards", however. (You can't literally add them back, but you can create a new iterator that gives you the consumed items first, and then continues giving the unconsumed items) – Aleksi Torhamo Nov 11 '18 at 09:25
Okay, thanks, luckily for my case I can get around it. I forgot about the `yield from`! Very nice solution. – Ross MacArthur Nov 11 '18 at 09:29

iGian · Answer 2 · 2018-11-11T10:12:40.043

0

Other option using zip_longest from itertools. It returns also true or false if all lists are consumed. Maybe not the most efficient way, but could be improved:

from itertools import zip_longest
a = ['a', 'b', 'c', 'd']
b = [1, 2, 3, 4, 5]
c = ['aa', 'bb', 'cc', 'dd', 'ee', 'ff']

def myzip(*iterables):
    consumed = True
    zips = []
    for zipped in zip_longest(*iterables):
      if None in zipped:
        consumed = False 
      else:
        zips.append(zipped)
    return [zips, consumed]


list(myzip(a, b, c))
#=> [[('a', 1, 'aa'), ('b', 2, 'bb'), ('c', 3, 'cc'), ('d', 4, 'dd')], False]

edited Nov 11 '18 at 10:12

answered Nov 11 '18 at 08:58

iGian

11,023
3
21
36

Please see edit to the question. Added an "Important Note". – Ross MacArthur Nov 11 '18 at 09:00
Can you provide a chunk of a real context object? I supposed it was a list. – iGian Nov 11 '18 at 09:05
Try using `iter(['a', 'b', 'c'])` and `iter([1, 2, 3, 4])` (obviously they can only be used once) – Ross MacArthur Nov 11 '18 at 09:10
I don't know if I got the point, but I made an edit based on your update. – iGian Nov 11 '18 at 09:27
I don't want it to end at the shortest, I want it to end at the rightmost. – Ross MacArthur Nov 11 '18 at 09:28
Ok, I'm late now, but I edited picking up your last reply. – iGian Nov 11 '18 at 10:14

score 0 · Answer 3 · answered Nov 11 '18 at 09:14

I think this one does the work by checking if the last consumer was completely consumed before returning

# Example copied from https://stackoverflow.com/questions/19151/build-a-basic-python-iterator
class Counter:
    def __init__(self, low, high):
        self.current = low
        self.high = high

    def __iter__(self):
        return self

    def __next__(self): # Python 3: def __next__(self)
        if self.current > self.high:
            raise StopIteration
        else:
            self.current += 1
            return self.current - 1

# modified from https://docs.python.org/3.5/library/functions.html#zip
def myzip(*iterables):
    sentinel = object()
    iterators = [iter(it) for it in iterables]
    while iterators:
        result = []
        for it in iterators:
            elem = next(it, sentinel)
            if elem is sentinel:
                elem = next(iterators[-1], sentinel)
                if elem is not sentinel:
                    raise ValueError("rightmost iterable was not consumed")
                else:
                    return
            result.append(elem)
        yield tuple(result)


a = Counter(1,7)
b = range(9)

for val in myzip(a,b):
    print(val)

Patrick Artner · Answer 4 · 2018-11-11T09:24:58.067

There is already a zip_longest in itertools that allows for "expansion" of the shorter iterable by a default value.

Use that and check if your default value occurs: if so, it would have been a case of "rightmost element not consumed":

class MyError(ValueError):
    """Unique "default" value that is recognizeable and allows None to be in your values.""" 
    pass

from itertools import zip_longest

isMyError = lambda x:isinstance(x,MyError)

def myzip(a,b):
    """Raises MyError if any non-consumed elements would occur using default zip()."""
    K = zip_longest(a,b, fillvalue=MyError())
    if all(not isMyError(t) for q in K for t in q): 
        return zip(a,b)
    raise MyError("Not all items are consumed") 


a = ['a', 'b', 'c', 'd']
b = [1, 2, 3, 4]
f = myzip(a, b)
print(list(f)) 
try:
    a = ['a', 'b', ]
    b = [1, 2, 3, 4]
    f = myzip(a, b)
    print(list(f)) 
except MyError as e:
    print(e)

Output:

[('a', 1), ('b', 2), ('c', 3), ('d', 4)]
Not all items are consumed

This consumes (worst case) the full zipped list once to check and then returns it as iterable.

zip-like function that fails if a particular iterator is not consumed

4 Answers4