python -- callable iterator size?

Question

I am looking through some text file for a certain string with the method.

re.finditer(pattern,text) I would like to know when this returns nothing. meaning that it could find nothing in the passed text.

I know that callable iterators, have next() and __iter__

I would like to know if I could get the size or find out if it returns no string matching my pattern.

Possible duplicate: http://stackoverflow.com/questions/3345785/getting-number-of-elements-in-an-iterator-in-python/ — Daenyth, Jul 27 '10 at 19:28
If you paste the code that you are working with, we might be able to come up with better answers. — Hamish Grubijan, Jul 28 '10 at 14:28

hynekcer · Answer 1 · 2019-01-27T04:07:06.927

This solution uses less memory, because it does not save intermediate results, as do other solutions that use list:

sum(1 for _ in re.finditer(pattern, text))

All older solutions have the disadvantage of consuming a lot of memory if the pattern is very frequent in the text, like pattern '[a-z]'.

Test case:

pattern = 'a'
text = 10240000 * 'a'

This solution with sum(1 for ...) uses approximately only the memory for the text as such, that is len(text) bytes. The previous solutions with list can use approximately 58 or 110 times more memory than is necessary. It is 580 MB for 32-bit resp. 1.1 GB for 64-bit Python 2.7.

Hamish Grubijan · Accepted Answer · 2012-06-11T16:21:02.407

7

EDIT 3: The answer by @hynekcer is much much better than this.

EDIT 2: This will not work if you have an infinite iterator, or one which consumes too many Gigabytes (in 2010 1 Gigabyte is still a large amount of ram/ disk space) of RAM/disk space.

You have already seen a good answer, but here is an expensive hack that you can use if you want to eat a cake and have it too :) The trick is that we have to clone the cake, and when you are done eating, we put it back into the same box. Remember, when you iterate over the iterator, it usually becomes empty, or at least loses previously returned values.

>>> def getIterLength(iterator):
    temp = list(iterator)
    result = len(temp)
    iterator = iter(temp)
    return result

>>>
>>> f = xrange(20)
>>> f
xrange(20)
>>> 
>>> x = getIterLength(f)
>>> x
20
>>> f
xrange(20)
>>>

EDIT: Here is a safer version, but using it still requires some discipline. It does not feel quite Pythonic. You would get the best solution if you posted the whole relevant code sample that you are trying to implement.

>>> def getIterLenAndIter(iterator):
    temp = list(iterator)
    return len(temp), iter(temp)

>>> f = iter([1,2,3,7,8,9])
>>> f
<listiterator object at 0x02782890>
>>> l, f = getIterLenAndIter(f)
>>> 
>>> l
6
>>> f
<listiterator object at 0x02782610>
>>>

edited Jun 11 '12 at 16:21

answered Jul 27 '10 at 19:32

Hamish Grubijan

10,562
23
99
147

This doesn't work with most iterators or generators. `getIterLength` will consume your `iterator`; assigning `iter(temp)` to `iterator` inside the function only creates a new local variable called `iterator` there which is discarded upon return from the function. Try substituting the line `f = xrange(20)` in your example with `f = iter([1,2,3,4,5])` to see what I mean. – Tim Pietzcker Jul 27 '10 at 20:08
Or compare `id(f)` with `id(iterator)` at the start of the function (they are the same), `id(iterator)` at the end of the function (it's different) and `id(f)` upon return from the function (it's the same as before). You're not putting the cloned cake into the same box, you're putting it into a new one and throwing it away. – Tim Pietzcker Jul 27 '10 at 20:14
Interesting, though, that it does work with `xrange()`. It definitely doesn't work with `re.finditer()`. – Tim Pietzcker Jul 27 '10 at 20:26
1

I do not think my answer was good enough to be an accepted one. I clearly indicated that this is an expensive hack. Apparently it does not always work, although I am not convinced that it is broken either. I will re-work the solution to return the iterator. – Hamish Grubijan Jul 27 '10 at 21:01
@Tim Pietzcker - is the new version broken with `re.finditer()` as well? – Hamish Grubijan Jul 27 '10 at 21:09
It looks a lot better now. I'm not sure if it will work in all cases because you're changing the iterator from whatever type it is (callable_iterator, generator, file object etc.) to a list_iterator. But I haven't found a case yet where it breaks, so it's definitely worth a try - if there really is a valid use case for using an iterator *and* wanting to know its length beforehand. – Tim Pietzcker Jul 28 '10 at 06:25

score 5 · Answer 3 · answered Jul 27 '10 at 19:25

5

Nope sorry iterators are not meant to know length they just know what's next which makes them very efficient at going through Collections. Although they are faster they do no allow for indexing which including knowing the length of a collection.

answered Jul 27 '10 at 19:25

Jesus Ramos

22,940
10
58
88

1

+1. Iterators wouldn't be 1/5 as useful as they are if they were nailed to some length in advance. Use (any collection) for that. – Jul 27 '10 at 19:28
there is no way of knowing length unless you iterate through the whole sequence. – Jesus Ramos Jul 27 '10 at 19:29
iterators are just for efficiency and should generally be used if you need to go through an entire collection regardless of order, it's always faster to iterate through an array or collection with an iterator than increment an index and check each index. – Jesus Ramos Jul 27 '10 at 19:31

score 1 · Answer 4 · answered Jul 27 '10 at 19:27

You can get the number of elements in an iterator by doing:

len( [m for m in re.finditer(pattern, text) ] )

Iterators are iterators because they have not generated the sequence yet. This above code is basically extracting each item from the iterator until it wants to stop into a list, then taking the length of that array. Something that would be more memory efficient would be:

count = 0
for item in re.finditer(pattern, text):
    count += 1

A tricky approach to the for-loop is to use reduce to effectively count the items in the iterator one by one. This is effectively the same thing as the for loop:

reduce( (lambda x, y : x + 1), myiterator, 0)

This basically ignores the y passed into reduce and just adds one. It initializes the running sum to 0.

score 1 · Answer 5 · answered Mar 24 '16 at 21:25

While some iterators might be able to know their length (for example, they were created from a string or a list) most do not and cannot. re.iter is a good example of one that cannot know it's length until it is finished.

However, there are a couple different ways to improve your current code:

use re.search to find if there are any matches, then use re.finditer to do the actual processing; or
use a sentinel value with the for loop.

The second option looks something like:

match = empty = object()
for match in re.finditer(...):
    # do some stuff
if match is empty:
    # there were no matches

score 0 · Answer 6 · answered Jul 27 '10 at 19:28

A quick solution would be to turn your iterator into a list and check the length of that list, but doing so can be bad for memory if there are too many results.

matches = list(re.finditer(pattern,text))
if matches:
  do_something()
print("Found",len(matches),"matches")

python -- callable iterator size?

6 Answers6