How come regex match objects aren't iterable even though they implement getitem?

Question

As you may know, implementing a __getitem__ method makes a class iterable:

class IterableDemo:
    def __getitem__(self, index):
        if index > 3:
            raise IndexError

        return index

demo = IterableDemo()
print(demo[2])  # 2
print(list(demo))  # [0, 1, 2, 3]
print(hasattr(demo, '__iter__'))  # False

However, this doesn't hold true for regex match objects:

>>> import re
>>> match = re.match('(ab)c', 'abc')
>>> match[0]
'abc'
>>> match[1]
'ab'
>>> list(match)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
TypeError: '_sre.SRE_Match' object is not iterable

It's worth noting that this exception isn't thrown in the __iter__ method, because that method isn't even implemented:

>>> hasattr(match, '__iter__')
False

So, how is it possible to implement __getitem__ without making the class iterable?

@Sweeper `Match.__getitem__` was added in Python 3.6 https://docs.python.org/3/library/re.html#re.Match.__getitem__ — DeepSpace, Sep 16 '18 at 19:28
Possible duplicate of [TypeError: '\_sre.SRE\_Match' object has no attribute '\_\_getitem\_\_'](https://stackoverflow.com/questions/22313702/typeerror-sre-sre-match-object-has-no-attribute-getitem) — Pedro Lobito, Sep 16 '18 at 19:29
Maybe helpful: https://github.com/python/cpython/blob/7f0d59f3a83cf1a98398b8e5bdfb97c7a71216bd/Modules/_sre.c#L1329 — Andrew Li, Sep 16 '18 at 19:30
@PedroLobito That's a completely different question, isn't it? My match objects **do** have a `__getitem__` method... — Aran-Fey, Sep 16 '18 at 19:31
@Li357 Also https://github.com/python/cpython/blob/e42b705188271da108de42b55d9344642170aa2b/Modules/sre.h#L44-L53 — DeepSpace, Sep 16 '18 at 19:34

Antti Haapala -- Слава Україні · Accepted Answer · 2018-09-23T09:54:53.587

There are lies, damned lies and then there is Python documentation.

Having __getitem__ for a class implemented in C is not enough for it to be iterable. That is because there are actually 2 places in the PyTypeObject where the __getitem__ can be mapped to: tp_as_sequence and tp_as_mapping. Both have a slot for __getitem__ ([1], [2]).

Looking at the source of the SRE_Match, tp_as_sequence is initialized to NULL whereas tp_as_mapping is defined.

The iter() built-in function, if called with one argument, will call the PyObject_GetIter, which has the following code:

f = t->tp_iter;
if (f == NULL) {
    if (PySequence_Check(o))
        return PySeqIter_New(o);
    return type_error("'%.200s' object is not iterable", o);
}

It first checks the tp_iter slot (obviously NULL for _SRE_Match objects); and failing that, then if PySequence_Check returns true, a new sequence iterator, else a TypeError is raised.

PySequenceCheck first checks if the object is a dict or a dict subclass - and returns false in that case. Otherwise it returns the value of

s->ob_type->tp_as_sequence &&
    s->ob_type->tp_as_sequence->sq_item != NULL;

and since s->ob_type->tp_as_sequence was NULL for a _SRE_Match instance, 0 will be returned, and PyObject_GetIter raises TypeError: '_sre.SRE_Match' object is not iterable.

How come regex match objects aren't iterable even though they implement __getitem__?

1 Answers1

How come regex match objects aren't iterable even though they implement getitem?