How do I check if an iterator is actually an iterator container?

Question

I have a dummy example of an iterator container below (the real one reads a file too large to fit in memory):

class DummyIterator:
    def __init__(self, max_value):
        self.max_value = max_value

    def __iter__(self):
        for i in range(self.max_value):
            yield i

def regular_dummy_iterator(max_value):
    for i in range(max_value):
        yield i

This allows me to iterate over the value more than once so that I can implement something like this:

def normalise(data):
    total = sum(i for i in data)
    for val in data:
        yield val / total

# this works when I call next()
normalise(DummyIterator(100))

# this doesn't work when I call next()
normalise(regular_dummy_iterator(100))

How do I check in the normalise function that I am being passed an iterator container rather than a normal generator?

You don't have to read the file into memory at once so why do you need this? — Padraic Cunningham, Jul 06 '15 at 12:08
What Padraic said. If you need to perform multiple passes over a large file just use its `.seek()` method to rewind it. — PM 2Ring, Jul 06 '15 at 12:16
Can you edit your code to make it runnable? Add `self` parameter to `__init__` and use `self.max_value` into `__iter__` — 301_Moved_Permanently, Jul 06 '15 at 12:23

score 9 · Accepted Answer · answered Jul 06 '15 at 12:03

First of all: There is no such thing as a iterator container. You have an iterable.

An iterable produces an iterator. Any iterator is also an iterable, but produces itself as the iterator:

>>> list_iter = iter([])
>>> iter(list_iter) is list_iter
True

You don't have an iterator if the iter(ob) is ob test is false.

score 3 · Answer 2 · edited May 23 '17 at 11:51

You can test whether you have an iterator (is consumed once next raises the StopIteration exception) vs just an iterable (can probably be iterated over multiple times) by using the collections.abcmodule. Here is an example:

from collections.abc import Iterable, Iterator

def my_iterator(): 
    yield 1

i = my_iterator()
a = []

isinstance(i, Iterator) # True
isinstance(a, Iterator) # False

What makes my_iterator() an Iterator is the presence of both the __next__ and __iter__ magic methods (and by the way, basically what is happening behind the scenes when you call isinstance on a collections.abc abstract base class is a test for the presence of certain magic methods).

Note that an iterator is also an Iterable, as is the empty list (i.e., both have the __iter__ magic method):

isinstance(i, Iterable) # True
isinstance(a, Iterable) # True

Also note, as was pointed out in Martijn Pieters' answer, that when you apply the generic iter() function to both, you get an iterator:

isinstance(iter(my_iterator()), Iterator) # True
isinstance(iter([])), Iterator) # True

The difference here between [] and my_iterator() is that iter(my_iterator()) returns itself as the iterator, whereas iter([]) produces a new iterator every time.

As was already mentioned in MP's same answer, your object above is not an "iterator container." It is an iterable object, i.e., "an iterable". Whether or not it "contains" something isn't really related; the concept of containing is represented by the abstract base class Container. A Container may be iterable, but it doesn't necessarily have to be.

How do I check if an iterator is actually an iterator container?

2 Answers2

Linked