Why does a class need __iter__() to return an iterator?

Question

Why does a class need to define __iter__() returning self, to get an iterator of the class?

class MyClass:
    def __init__(self):
        self.state = 0

    def __next__(self):
        self.state += 1
        if self.state > 4:
            raise StopIteration
        return self.state

myObj = MyClass()
for i in myObj:
    print(i)

Console log:

Traceback (most recent call last):
   for i in myObj:
TypeError: 'MyClass' object is not iterable

the answer https://stackoverflow.com/a/9884259/4515198, says

An iterator is an object with a next (Python 2) or __next__ (Python 3) method.

The task of adding the following:

def __iter__(self):
   return self

is to return an iterator, or an object of the class, which defines the __next__() method.

But, isn't the task of returning an object of MyClass (which defines the __next__() method) already done by the __new__() method, when MyClass is instantiated in the line myObj = MyClass() ?

Won't the objects of a class defining __next__() method, be iterators by themselves?

I have studied the questions What is the use of returning self in the __iter__ method? and Build a Basic Python Iterator, but I am still unable to understand the reason for having an __iter__() method returning self.

Answer 1

The answer to the question of why the __iter__() method is necessary is that for for-loops always start by calling iter() on an object to get an iterator. That is why even iterators themselved need an __iter__() method to work with for-loops. After for calls iter(), then it calls __next__() on the resulting iterator to obtain a value.

The rules for creating iterables and iterators are:

1) Iterables have an __iter__() method that returns an iterator.

2) Iterators have a __next__() method that returns a value, that updates the state, and that raises StopIteration when complete.

3) Iterators themselves have a __iter__() method that returns self. That means that all iterators are self-iterable.

The benefit of the last rule for iterators having an __iter__() method that returns self is that it allows us to pass around partially consumed iterators:

>>> s = 'hello world'
>>> it = iter(s)
>>> next(it)
'h'
>>> next(it)
'e'
>>> list(it)     # Doesn't start from the beginning
['l', 'l', 'o', ' ', 'w', 'o', 'r', 'l', 'd']

Here's another example that depends on iterators being self-iterable without restarting:

>>> s = 'hello world'
>>> it = iter(s)
>>> list(zip(it, it))
[('h', 'e'), ('l', 'l'), ('o', ' '), ('w', 'o'), ('r', 'l')]

Notes:

1) An alternative way to make an iterable is to supply a __getitem__() method that accepts consecutive indices and raises IndexError when complete. This is how str objects iterated in Python 2.

2) Some objects like files are their own iterator. That means that you can call next() directly on a file object. It also means that files cannot have multiple, independent iterators (the file object itself has the state tracking the position within the file).

3) The iterator design pattern described above isn't Python specific. It is a general purpose design pattern for many OOP languages: https://en.wikipedia.org/wiki/Iterator_pattern

Answer 2

isn't that already done by the __new__() method

Nope, __new__ is just another method, it doesn't automatically create __iter__ for an object.

Won't the objects of a class defining __next__() method, be iterators by themselves?

Not necessarily, as your first class defining __next__ but not __iter__ shows. __next__ is needed if you need to support iteration since it produces the values. __iter__ is needed since that's what's called on an object in the for statement in order for it to get an iterator.

You could have a class that only defines __next__ and somewhat works (it is limited) but, it needs to be returned from someones __iter__ . For example, the class MyClass returns a class CustomIter that only defines __next__:

class MyClass:
    def __iter__(self): 
        return CustomIter()

class CustomIter(object):

    def __init__(self): 
        self.state = 0

    def __next__(self): 
        self.state += 1
        if self.state > 4:
            raise StopIteration
        return self.state

You'll need an __iter__ defined on an object that will return another object (could be itself) on which __next__ is defined.

---

If your class defines __iter__ as:

def __iter__(self): return self

then you need to define __next__ on type(self) (the class) since you return the instance itself. __next__ is going to get called on self until no more values can be produced.

The other case is when __iter__ simply returns another object which defines __next__ (as per my first example). You can alternatively do that by making __iter__ a generator.

For example:

class MyClass:
    def __init__(self):
        self.state = 0

    def __iter__(self): 
        for i in range(10): yield i

doesn't define a __next__. When iter is called on it though:

g = iter(MyClass())

it returns a generator g that defines __next__:

g = iter(MyClass())

g.__next__() # 0
g.__next__() # 1