Monkey patching operator overloads behaves differently in Python2 vs Python3

Question

Consider the following code:

class Foo:
    def __mul__(self,other):
        return other/0
x = Foo()
x.__mul__ = lambda other:other*0.5
print(x.__mul__(5))
print(x*5)

In Python2 (with from future import print), this outputs

2.5
2.5

In Python3, this outputs

2.5

---------------------------------------------------------------------------
ZeroDivisionError                         Traceback (most recent call last)
<ipython-input-1-36322c94fe3a> in <module>()
      5 x.__mul__ = lambda other:other*0.5
      6 print(x.__mul__(5))
----> 7 print(x*5)

<ipython-input-1-36322c94fe3a> in __mul__(self, other)
      1 class Foo:
      2     def __mul__(self,other):
----> 3         return other/0
      4 x = Foo()
      5 x.__mul__ = lambda other:other*0.5

ZeroDivisionError: division by zero

I ran into this situation when I was trying to implement a type that supported a subset of algebraic operations. For one instance, I needed to modify the multiplication function for laziness: some computation must be deferred until the instance is multiplied with another variable. The monkey patch worked in Python 2, but I noticed it failed in 3.

Why does this happen? Is there any way to get more flexible operator overloading in Python3?

score 2 · Accepted Answer · answered Oct 01 '16 at 01:17

That is not a monkeypatch.

This would have been a monkeypatch:

class Foo:
    def __mul__(self, other):
        return other / 0

Foo.__mul__ = lambda self,other: other * 0.5

x = Foo()
x*9    # prints 4.5

What was done with x.__mul__ = lambda other:other*0.5 was creating a __mul__ attribute on the x instance.

Then, it was expected that x*5 would call x.__mul__(5). And it did, in Python 2.

In Python 3, it called Foo.__mul__(x, 5), so the attribute was not used.

Python 2 would have done the same as Python 3, but it did not because Foo was created as an old-style class.

This code would be equivalent for Python 2 and Python 3:

class Foo(object):
    def __mul__(self,other):
        return other/0
x = Foo()
x.__mul__ = lambda other:other*0.5
print(x.__mul__(5))
print(x*5)

That will raise an exception. Note the (object).

score 1 · Answer 2 · edited Jun 20 '20 at 09:12

You can't override the special methods on instance level. Based on python's documentation:

For custom classes, implicit invocations of special methods are only guaranteed to work correctly if defined on an object’s type, not in the object’s instance dictionary.

The rationale behind this behaviour lies with a number of special methods such as __hash__() and __repr__() that are implemented by all objects, including type objects. If the implicit lookup of these methods used the conventional lookup process, they would fail when invoked on the type object itself:

>>> 1 .__hash__() == hash(1)
True
>>> int.__hash__() == hash(int)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
TypeError: descriptor '__hash__' of 'int' object needs an argument

So one simple way is defining a regular function for your your monkey-patching aims, and assign your new method to it:

In [45]: class Foo:
             def __init__(self, arg):
                 self.arg = arg
             def __mul__(self,other):
                 return other * self.arg
             def _mul(self, other):
                 return other/0

Demo:

In [47]: x = Foo(10)

In [48]: x * 3
Out[48]: 30

In [49]: my_func = lambda x: x * 0.5

In [50]: x._mul = my
my_func  mypub/   

In [50]: x._mul = my_func

In [51]: x._mul(4)
Out[51]: 2.0

Monkey patching operator overloads behaves differently in Python2 vs Python3

2 Answers2