19

What's the result of returning NotImplemented from __eq__ special method in python 3 (well 3.5 if it matters)?

The documentation isn't clear; the only relevant text I found only vaguely refers to "some other fallback":

When NotImplemented is returned, the interpreter will then try the reflected operation on the other type, or some other fallback, depending on the operator. If all attempted operations return NotImplemented, the interpreter will raise an appropriate exception. See Implementing the arithmetic operations for more details.

Unfortunately, the "more details" link doesn't mention __eq__ at all.

My reading of this excerpt suggests that the code below should raise an "appropriate exception", but it does not:

class A:
  def __eq__(self, other):
    return NotImplemented

class B:
  def __eq__(self, other):
    return NotImplemented

# docs seems to say these lines should raise "an appropriate exception"
# but no exception is raised
a = A()
b = B()
a == b # evaluates as unequal
a == a # evaluates as equal

From experimenting, I think that when NotImplemented is returned from __eq__, the interpreter behaves as if __eq__ wasn't defined in the first place (specifically, it first swaps the arguments, and if that doesn't resolve the issue, it compares using the default __eq__ that evaluates "equal" if the two objects have the same identity). If that's the case, where in the documentation can I find the confirmation of this behavior?

Edit: see Python issue 28785

MSeifert
  • 145,886
  • 38
  • 333
  • 352
max
  • 49,282
  • 56
  • 208
  • 355
  • 1
    to raise exception you will need `raise NotImplementedError` in your code. – furas Nov 24 '16 at 07:19
  • in my Python 3 both evaluate as unequal. Put `print("A.eq")` and `print("A.eq")` and see what's happen. First is called "A.eq", later "B.eq". And later probably is called `eq` in other data type which can compares it and returns result - so it doesn't raise error. Probably it compares `id(A())` and `id(B())`. – furas Nov 24 '16 at 07:26
  • `id()` can work with every objects so you don't get error. `__add__` doesn't have some universal method which always works so it can raise exception. – furas Nov 24 '16 at 07:32
  • I don't have link to documentation but I think it finally does `id(a) == id(b)`, `id(a) == id(a)` – furas Nov 24 '16 at 07:41
  • 1
    http://stackoverflow.com/questions/878943/why-return-notimplemented-instead-of-raising-notimplementederror?rq=1 - first answer confirms that it uses identity. I'm not sure about relevant interpreter code snippet or official docs link. – Łukasz Rogalski Nov 24 '16 at 07:50
  • Page 399 of **Fluent Python** (O'Reilly, 2015) provides a table that shows that the fallback of `a==b` is `id(a)==id(b)`, although I cannot find such in the official documentation. I would provide this table as an answer, but whereas most O'Reilly books have a disclaimer giving permission to use excerpts for that purpose, this one seems to not have it. Without that explicitly allowed, I don't want to use the excerpt (although it may fall under fair use). – Matthew Nov 24 '16 at 07:54
  • 1
    @ŁukaszRogalski makes sense, but the link is about python 2 as of 2009, and is based on a blog post rather than documentation. Shouldn't it be in the official python 3 documentation? Maybe I should make an issue on bugs.python.org? – max Nov 24 '16 at 07:55
  • @max, I agree that it should be in the official documentation, but seems not to be (or if it is, it is buried somewhere far away from the docs on the rich comparison operators). I'd never considered your question, and once you asked it, as they stand the official docs are useless to answer it. – Matthew Nov 24 '16 at 08:01
  • @max sure, it definitely is _quirky_ behavior and it would be helpful if this will be included in docs. If you feel that's what you want to do (submit a bug, maybe even a pull request), go for it. – Łukasz Rogalski Nov 24 '16 at 08:05
  • As far as I can tell, it's not addressed in the Python 3 documentation. But [it is](https://docs.python.org/2/reference/datamodel.html#object.__cmp__) documented in Python 2 on the `__cmp__()` function (which was removed in 3 and the documentation along with it). `object.__cmp__: Called by comparison operations if rich comparison (see above) is not defined... If no __cmp__(), __eq__() or __ne__() operation is defined, class instances are compared by object identity ("address").` – Jeff Mercado Nov 24 '16 at 08:06

2 Answers2

17

Actually the == and != check work identical to the ordering comparison operators (< and similar) except that they don't raise the appropriate exception but fall-back to identity comparison. That's the only difference.

This can be easily seen in the CPython source code (version 3.5.10). I will include a Python version of that source code (at least as far as it's possible):

_mirrored_op = {'__eq__': '__eq__',  # a == b => b == a
                '__ne__': '__ne__',  # a != b => b != a
                '__lt__': '__gt__',  # a < b  => b > a
                '__le__': '__ge__',  # a <= b => b >= a
                '__ge__': '__le__',  # a >= b => b <= a
                '__gt__': '__lt__'   # a > b  => b < a
               }

def richcmp(v, w, op):
    checked_reverse = 0
    # If the second operand is a true subclass of the first one start with
    # a reversed operation.
    if type(v) != type(w) and issubclass(type(w), type(v)) and hasattr(w, op):
        checked_reverse = 1
        res = getattr(w, _mirrored_op[op])(v)     # reversed
        if res is not NotImplemented:
            return res
    # Always try the not-reversed operation
    if hasattr(v, op):
        res = getattr(v, op)(w)      # normal
        if res is not NotImplemented:
            return res
    # If we haven't already tried the reversed operation try it now!
    if not checked_reverse and hasattr(w, op):
        res = getattr(w, _mirrored_op[op])(v)      # reversed
        if res is not NotImplemented:
            return res
    # Raise exception for ordering comparisons but use object identity in 
    # case we compare for equality or inequality
    if op == '__eq__':
        res = v is w
    elif op == '__ne__':
        res = v is not w
    else:
        raise TypeError('some error message')

    return res

and calling a == b then evaluates as richcmp(a, b, '__eq__'). The if op == '__eq__' is the special case that makes your a == b return False (because they aren't identical objects) and your a == a return True (because they are).

However the behavior in Python 2.x was completely different. You could have up to 4 (or even 6, I don't remember exactly) comparisons before falling back to identity comparison!

MSeifert
  • 145,886
  • 38
  • 333
  • 352
10

Not sure where (or if) it is in the docs, but the basic behavior is:

  • try the operation: __eq__(lhs, rhs)
  • if result is not NotImplemented return it
  • else try the reflected operation: __eq__(rhs, lhs)
  • if result is not NotImplemented return it
  • otherwise use appropriate fall back:

eq -> same objects? -> True, else False

ne -> different objects? -> True, else False

many others -> raise exception

The reason that eq and ne do not raise exceptions is:

  • they can always be determined (apple == orange? no)
Antony Hatchkins
  • 31,947
  • 10
  • 111
  • 111
Ethan Furman
  • 63,992
  • 20
  • 159
  • 237
  • Ahh ... Ok, I was confused when you talked about "same objects" in your comment. The new (sticky) header on StackOverflow blocked me from seeing that you'd edited the post. Oops. – mgilson Mar 03 '17 at 21:41
  • 1
    That strikes me as an incredibly bad design decision. Never mind that it causes hard to catch bugs, but the rationale makes no sense whatsoever: when an implementation determines that two objects are not comparable (`NotImplemented`), why would object identity *every* yield `True`? This is simply never the case, unless the implementation of `__eq__` is broken at the outset. – Konrad Rudolph Jul 22 '20 at 12:03
  • @KonradRudolph: I do not understand your comment about "identify *every* yield `True`" -- did a word get dropped? – Ethan Furman Jul 22 '20 at 15:11
  • @Ethan I meant “ever”, not “every”. – Konrad Rudolph Jul 22 '20 at 15:13
  • @KonradRudolph: `NotImplemented` means "did not bother to implement anything", not "impossible to know". object identity will return `True` if the `lhs` and `rhs` are the same object. – Ethan Furman Jul 22 '20 at 15:51
  • @EthanFurman I’m aware what it means, but that doesn’t change my comment. *In practice*, can you show me a situation where this will ever lead to an object identity being `True` without there being a logic error in the `__eq__` implementation? I maintain that this situation does not exist (or, if it does, only as an esoteric corner case with no practical relevance). My point is: rather than test for object identity, the fallback could just be hard-coded to `False` (though to be honest I’d **strongly** prefer an error being raised, since the current behaviour manifestly leads to bugs). – Konrad Rudolph Jul 22 '20 at 16:24
  • I think what @Konrad is saying is that, if `__eq__` is implemented correctly, it will never be the case that `a.__eq__(a)` will be `NotImplemented` (what kind of equality doesn't implement comparing an object to itself?), thus there is no point in falling back to identity comparison because if both `a.__eq__(b)` and `b.__eq__(a)` are `NotImplemented`, it cannot (should not) be the case that `a is b`, so this fallback will always return `False`. – Anakhand Dec 15 '20 at 13:55
  • But I think this decision can be defended: hardcoding `False` would cause a bad implementation of `__eq__` as described above to cause hard to find bugs (equality would not be reflexive), and raising an error would cause problems when testing equality between two distinct objects of different types that both happen to return NotImplemented for this particular comparison; raising an error when testing equality would be a bad idea in general because in principle it should always be possible to implement equality. – Anakhand Dec 15 '20 at 13:59
  • 1
    I think the bad design decision here was the `NotImplemented` concept itself (at least for `__eq__` and `__ne__`): the author of a comparison method should always explicitly select a fallback comparison (if any); using an implicit fallback that changes depending on the comparison and the pair of objects is what leads to all of this. – Anakhand Dec 15 '20 at 14:02