Why is NotImplemented evaluated multiple times with __eq__ operator

Question

Don’t mix up apples and oranges

The problem

I’m playing with the __eq__ operator and the NotImplemented value.

I’m trying to understand what’s happen when obj1.__eq__(obj2) returns NotImplemented and obj2.__eq__(obj1) also returns NotImplemented.

According to the answer to Why return NotImplemented instead of raising NotImplementedError, and the detailed article How to override comparison operators in Python in the "LiveJournal" blog, the runtime should fall back to the built-in behavior (which is based on identity for == and !=).

Code sample

But, trying the example bellow, it seems that I have multiple calls to __eq__ for each pair of objects.

class Apple(object):
    def __init__(self, color):
        self.color = color

    def __repr__(self):
        return "<Apple color='{color}'>".format(color=self.color)

    def __eq__(self, other):
        if isinstance(other, Apple):
            print("{self} == {other} -> OK".format(self=self, other=other))
            return self.color == other.color
        print("{self} == {other} -> NotImplemented".format(self=self, other=other))
        return NotImplemented


class Orange(object):
    def __init__(self, usage):
        self.usage = usage

    def __repr__(self):
        return "<Orange usage='{usage}'>".format(usage=self.usage)

    def __eq__(self, other):
        if isinstance(other, Orange):
            print("{self} == {other}".format(self=self, other=other))
            return self.usage == other.usage
        print("{self} == {other} -> NotImplemented".format(self=self, other=other))
        return NotImplemented

>>> apple = Apple("red")
>>> orange = Orange("juice")

>>> apple == orange
<Apple color='red'> == <Orange usage='juice'> -> NotImplemented
<Orange usage='juice'> == <Apple color='red'> -> NotImplemented
<Orange usage='juice'> == <Apple color='red'> -> NotImplemented
<Apple color='red'> == <Orange usage='juice'> -> NotImplemented
False

Expected behavior

I expected to have only:

<Apple color='red'> == <Orange usage='juice'> -> NotImplemented
<Orange usage='juice'> == <Apple color='red'> -> NotImplemented

Then falling back to identity comparison id(apple) == id(orange) -> False.

Answer 1

This is issue #6970 in the Python tracker; it remains unfixed in 2.7 and Python 3.0 and 3.1.

This is caused by two places trying both the straight and the swapped comparison when a comparison between two custom classes with __eq__ methods is executed.

Rich comparisons go through the PyObject_RichCompare() function, which for objects with different types (indirectly) delegates to try_rich_compare(). In this function v and w are the left and right operand objects, and since both have a __eq__ method the function calls both v->ob_type->tp_richcompare() and w->ob_type->tp_richcompare().

For custom classes, the tp_richcompare() slot is defined as the slot_tp_richcompare() function, and this function again executes __eq__ for both sides, first self.__eq__(self, other) then other.__eq__(other, self).

In the end, that means apple.__eq__(apple, orange) and orange.__eq__(orange, apple) is called for the first attempt in try_rich_compare(), and then the reverse is called, resulting in the orange.__eq__(orange, apple) and apple.__eq__(apple, orange) calls as self and other are swapped in slot_tp_richcompare().

Note that the issue is limited to instances of differing custom classes where both classes define an __eq__ method. If either side doesn't have such a method __eq__ is only executed once:

>>> class Pear(object):
...     def __init__(self, purpose):
...         self.purpose = purpose
...     def __repr__(self):
...         return "<Pear purpose='{purpose}'>".format(purpose=self.purpose)
...    
>>> pear = Pear("cooking")
>>> apple == pear
<Apple color='red'> == <Pear purpose='cooking'> -> NotImplemented
False
>>> pear == apple
<Apple color='red'> == <Pear purpose='cooking'> -> NotImplemented
False

If you have two instances of the same type and __eq__ returns NotImplemented, you get six comparisons even:

>>> class Kumquat(object):
...     def __init__(self, variety):
...         self.variety = variety
...     def __repr__(self):
...         return "<Kumquat variety=='{variety}'>".format(variety=self.variety)
...     def __eq__(self, other):
...         # Kumquats are a weird fruit, they don't want to be compared with anything
...         print("{self} == {other} -> NotImplemented".format(self=self, other=other))
...         return NotImplemented
...
>>> Kumquat('round') == Kumquat('oval')
<Kumquat variety=='round'> == <Kumquat variety=='oval'> -> NotImplemented
<Kumquat variety=='oval'> == <Kumquat variety=='round'> -> NotImplemented
<Kumquat variety=='round'> == <Kumquat variety=='oval'> -> NotImplemented
<Kumquat variety=='oval'> == <Kumquat variety=='round'> -> NotImplemented
<Kumquat variety=='oval'> == <Kumquat variety=='round'> -> NotImplemented
<Kumquat variety=='round'> == <Kumquat variety=='oval'> -> NotImplemented
False

The first set of two comparisons were called from an attempt at optimising; when two instances have the same type, you only need to call v->tp_richcompare(v, w) and coercions (for numbers) can be skipped, after all. However, when that comparison fails (NotImplemented is returned), then the standard path is also tried.

How comparisons are done in Python 2 got rather complicated as the older __cmp__ 3-way comparison method still had to be supported; in Python 3, with support for __cmp__ dropped, it was easier to fix the issue. As such, the fix was never backported to 2.7.