Python - Count Elements in Iterator Without Consuming

Question

Given an iterator it, I would like a function it_count that returns the count of elements that iterator produces, without destroying the iterator. For example:

ita = iter([1, 2, 3])
print(it_count(ita))
print(it_count(ita))

should print

3
3

It has been pointed out that this may not be a well-defined question for all iterators, so I am not looking for a completely general solution, but it should function as anticipated on the example given.

---

Okay, let me clarify further to my specific case. Given the following code:

ita = iter([1, 2, 3])
itb, itc = itertools.tee(ita)
print(sum(1 for _ in itb))
print(sum(1 for _ in itc))

...can we write the it_count function described above, so that it will function in this manner? Even if the answer to the question is "That cannot be done," that's still a perfectly valid answer. It doesn't make the question bad. And the proof that it is impossible would be far from trivial...

Answer 1

Not possible. Until the iterator has been completely consumed, it doesn't have a concrete element count.

Answer 2

The only way to get the length of an arbitary iterator is by iterating over it, so the basic question here is ill-defined. You can't get the length of any iterator without iterating over it.

Also the iterator itself may change it's contents while being iterated over, so the count may not be constant anyway.

---

But there are possibilities that might do what you ask, be warned none of them is foolproof or really efficient:

When using python 3.4 or later you can use operator.length_hint and hope the iterator supports it (be warned: not many iterators do! And it's only meant as a hint, the actual length might be different!):

>>> from operator import length_hint

>>> it_count = length_hint

>>> ita = iter([1, 2, 3])
>>> print(it_count(ita))
3
>>> print(it_count(ita))
3

---

As alternative: You can use itertools.tee but read the documentation of that carefully before using it. It may solve your issue but it won't really solve the underlying problem.

import itertools

def it_count(iterator):
    return sum(1 for _ in iterator)

ita = iter([1, 2, 3])
it1, it2 = itertools.tee(ita, 2)
print(it_count(it1))  # 3
print(it_count(it2))  # 3

But this is less efficient (memory and speed) than casting it to a list and using len on it.

Answer 3

There's no generic way to do what you want. An iterator may not have a well defined length (e.g. itertools.count which iterates forever). Or it might have a length that's expensive to calculate up front, so it won't let you know how far you have to go until you've reached the end (e.g. a file object, which can be iterated yielding lines, which are not easy to count without reading the whole file's contents).

Some kinds of iterators might implement a __length_hint__ method that returns an estimated length, but that length may not be accurate. And not all iterators will implement that method at all, so you probably can't rely upon it (it does work for list iterators, but not for many others).

Often the best way to deal with the whole contents of an iterator is to dump it into a list or other container. After you're done doing whatever operation you need (like calling len on it), you can iterate over the list again. Obviously this requires the iterator to be finite (and for all of its contents to fit into memory), but that's the limitation you have to deal with.

If you only need to peek ahead by a few elements, you might be able to use itertools.tee, but it's no better than dumping into a list if you need to consume the whole contents (since it keeps the values seen by one of its returned iterators but another in a data structure similar to a deque). It wouldn't be any use for finding the length of the iterator.

Answer 4

I have not been able to come up with an exact solution (because iterators may be immutable types), but here are my best attempts. I believe the second should be faster, according to the documentation (final paragraph of itertools.tee).

Option 1

def it_count(it):
   tmp_it, new_it = itertools.tee(it)
   return sum(1 for _ in tmp_it), new_it

Option 2

def it_count2(it):
   lst = list(it)
   return len(lst), lst

It functions well, but has the slight annoyance of returning the pair rather than simply the count.

ita = iter([1, 2, 3])
count, ita = it_count(ita)
print(count)

Output: 3

count, ita = it_count2(ita)
print(count)

Output: 3

count, ita = it_count(ita)
print(count)

Output: 3

print(list(ita))

Output: [1, 2, 3]