Why can `subclasshook` be monkeypatched onto the metaclass but `instancecheck` cannot?

Question

Here is a toy example of trying to create a decorator that allows declaration of attribute names which should be required parts of "interface checking" along the standard __subclasshook__ and __instancecheck__ patterns.

It seems to work as expected when I decorate the Foo class. I make a Bar class, unrelated to Foo, but which has the needed attributes, and it correctly satisfies isinstance(instance_of_bar, Foo) == True.

But then as another example, I make a subclass of dict augmented so that the key values will be accessible with getattr syntax as well (e.g. a dict where d['a'] can be replaced with d.a to get the same result). In this case, the attributes are just instance attributes, so __instancecheck__ should work.

Here is the code. Note that given that the example with the instance of Bar works, the choice to "monkeypatch" the __subclasshook__ function into the Foo class (that has a metaclass) works fine. So it does not seem that one must define the function directly in the class definition of the metaclass.

#Using Python 2.7.3

import abc
def interface(*attributes):
    def decorator(Base):

        def checker(Other):
            return all(hasattr(Other, a) for a in attributes)

        def __subclasshook__(cls, Other):
            if checker(Other):
                return True
            return NotImplemented

        def __instancecheck__(cls, Other):
            return checker(Other)

        Base.__subclasshook__ = classmethod(__subclasshook__)
        Base.__instancecheck__ = classmethod(__instancecheck__)
        return Base

    return decorator

@interface("x", "y")
class Foo(object):
    __metaclass__ = abc.ABCMeta
    def x(self): return 5
    def y(self): return 10

class Bar(object):
    def x(self): return "blah"
    def y(self): return "blah"

class Baz(object):
    def __init__(self):
        self.x = "blah"
        self.y = "blah"

class attrdict(dict):
    def __getattr__(self, attr):
        return self[attr]

f = Foo()
b = Bar()
z = Baz()
t = attrdict({"x":27.5, "y":37.5})

print isinstance(f, Foo)
print isinstance(b, Foo)
print isinstance(z, Foo)
print isinstance(t, Foo)

Prints:

True
True
False
False

This is just a toy example -- I'm not looking for better ways to implement my attrdict class. The Bar example demonstrates the monkeypatched __subclasshook__ working. The other two examples demonstrate failure of __instancecheck__ for instances that have mere instance attributes to check on. In those cases, __instancecheck__ is not even called.

I can manually check that the condition from my __instancecheck__ function is satisfied by the instance of attrdict (that is, hasattr(instance_of_attrdict, "x") is True as needed) or z.

Again, it seems to work fine for the instance of Bar. This suggests that __subclasshook__ is being correctly applied by the decorator, and that patching a custom __metaclass__ is not the issue. But __instancecheck__ does not seem to be called in the process.

Why can __subclasshook__ be defined outside of the class definition of the metaclass and added later, but not __instancecheck__?

This doesn't really have anything to do with built-in types. The `Other` parameter to `__subclasshook__` is the *class*, not an instance of the class. If you replaced `attrdict` with a class named `Baz` that assigned `self.x` and `self.y` in its `__init__` method, you'd get the exact same results because the class itself doesn't have those attributes, but an instance of it does. — Blender, Dec 11 '13 at 20:29
But why wouldn't `__subclasshook__` (by virtue of `hasattr(Other, a)` returning `False` since I incorrectly try to use `Other` like an instance instead of a class) just return `False` then, instead of the `AssertionError` that I am seeing. And the follow-up is then: how do I use `__subclasshook__` to define checking on whether *instances* happen to have the required attributes to be considered an instance, whether such attributes were monkeypatched, created as class attributes, created as instance attributes in `__init__` or anything else. — ely, Dec 11 '13 at 20:33
Your code shouldn't work in the first place, as `all()` returns a boolean, which has no `next()` method. The `AssertionError` just says that `__subclasshook__` is expecting the return type to be boolean, while you're returning something else. `__instancecheck__` is what you really should be implementing. — Blender, Dec 11 '13 at 20:46
Have you tried it? `all(...generator_expression...)` does work. — ely, Dec 11 '13 at 21:18
My original hope was the `all` would work around a generator expression the same way, e.g. `sum` does. I'm also puzzled by this. Why does `sum(x for x in [1,2,3])` return `6` whereas `all(x for x in [1,2,3])` returns a generator expression object? — ely, Dec 11 '13 at 21:20
Further, changing it to `all([hasattr(Other, a) for a in attributes])`, which definitely returns a `bool`, didn't help. — ely, Dec 11 '13 at 21:21
`all(x for x in [1,2,3])` returns `True` for me on both Python 2.7.5 and Python 3.3 — Blender, Dec 11 '13 at 21:21
I'm using 2.7.3 and it absolutely doesn't return `True` for me. It returns a generator object. — ely, Dec 11 '13 at 21:22
That's really strange and definitely should not be happening. What does `all is __builtin__.all` output? — Blender, Dec 11 '13 at 21:22
You are correct. I'm in an IPython interpreter and somehow `all` is masked by `` even though I meticulously guard my import statements and my IPython config files. I'm guessing it must be a property of the ipython binary managed by the IT team or something really crazy. — ely, Dec 11 '13 at 21:26
That fixes my other errors too. If I use `__builtin__.all`, then cases (1) and (2) return `True`. So now the question is just how to check whether instances themselves satisfy an interface. And it seems that `instancecheck` is the right method for that, so that `__subclasshook__` can be saved for just class-level checking? — ely, Dec 11 '13 at 21:30

score 2 · Accepted Answer · edited May 23 '17 at 10:27

Everything works as is should. If you are using __metaclass__, you are overwriting the class creation process. It looks like your monkeypatch is working for __subclasshook__ but it's only called from the __subclasshook__ function of the ABCMeta. You can check it with this:

>>> type(Foo)
<class 'abc.ABCMeta'>

To be more explicit: the __subclasshook__ case works by accident in this example, because the metaclass's __subclasscheck__ happens to defer to the class's __subclasshook__ in some situations. The __instancecheck__ protocol of the metaclass does not ever defer to the class's definition of __instancecheck__, which is why the monkeypatched version of __subclasshook__ does eventually get called, but the monkeypatched version of __instancecheck__ does not get called.

In more details: If you are creating a class with the metaclass, the type of the class will be the metaclass. In this case the ABCMeta. And the definition of the isinstance() say the following: 'isinstance(object, class-or-type-or-tuple) -> bool', which means the instance checking will be executed on the given class, type or tuple of classes/types. In this case the isinstance check will be done on the ABCMeta (ABCMeta.__instancecheck__() will be called). Because the monkeypatch was applied to the Foo class and not the ABCMeta, the __instancecheck__ method of Foo will never run. But the __instancecheck__ of the ABCMethod will call the __subclasscheck__ method of itself, and this second method will try the validaion by executing the __subclasshook__ method of created class (in this example Foo).

In this case you can get the desired behavior if you overwrite the functions of the metaclass like this:

def interface(*attributes):
    def decorator(Base):

        def checker(Other):
            return all(hasattr(Other, a) for a in attributes)

        def __subclasshook__(cls, Other):
            if checker(Other):
                return True
            return NotImplemented

        def __instancecheck__(cls, Other):
            return checker(Other)

        Base.__metaclass__.__subclasshook__ = classmethod(__subclasshook__)
        Base.__metaclass__.__instancecheck__ = classmethod(__instancecheck__)
        return Base

    return decorator

And the output with the updated code:

True
True
True
True

Another approach would be to define your own class to serve as the metaclass, and create the kind of __instancecheck__ protocol that you're looking for, so that it does defer to the class's definition of __instancecheck__ when the metaclass's definition hits some failure criteria. Then, set __metaclass__ to be that class inside of Foo and your existing decorator should work as-is.

More info: A good post about metaclasses

This is a good workaround, but not an answer to the question. If I were to write my class definition of `Foo` with "@classmethod ... def __instancecheck__(...)" then it works. Same for `__subclasshook__`. So defining these functions as classmethods *for `Foo` itself* **does** work. Monkeypatching them onto `Foo` also works -- for `__subclasshook__` (again, never needing to apply it to `__metaclass__` as you did). So, given all that, why doesn't the same procedure also work for `__instancecheck__`? In other words, there's a difference between the two with no documented reason to expect one. — ely, Dec 16 '13 at 17:40
This is not a workaround. In your example you are creating an instance of `ABCMeta` and not from `Foo` (you can see it in thy `type(Foo)` example. And because it's an instance of `ABCMeta`, it will act like the `ABCMeta`. You can see the source of it here: [link](http://hg.python.org/cpython/file/2.7/Lib/abc.py) The two method is defined here and when you are calling `isinstance`, this method will be executed. So if you want to overwrite the method, you have to overwrite/monkeypatch this and not the method of original `Foo` (because it's act like `Foo` but it's not the `Foo` itself). — Simon Sagi, Dec 16 '13 at 17:51
I believe that the fact that my `__subclasshook__` works after monkeypatching illustrates that it is correctly being defined as the *metaclass's* function, not "`Foo`s". I agree with you about printing `type(Foo)` and seeing `ABCMeta` -- that's not surprising to me; it is expected. I'm not sure what your point is about that. Even after re-reading your answer and comment, my question still remains. — ely, Dec 16 '13 at 18:00
The `ABCMeta` has 2 (relevant for now) function: `__instancecheck__` and `__subclasscheck__`. But you can see: `__subclasscheck__` != `__subclasshook__`. You can define `__subclasshook__` function for `Foo` and this `__subclasshook__` will be called from `__subclasscheck__` (Line no. 161). You are monkeypatching the `Foo` but not the function what is called (`__subclasscheck__`) because the called method is in the `ABCMeta` class. And still: you cannot overwrite the `__instancecheck__` method of `ABCMeta` from `Foo`. — Simon Sagi, Dec 16 '13 at 18:08
Ok, that makes sense. So the difference is that in the `__instancecheck__` protocol, it never has a failure mode where it defers to that subclass's definition of `__instancecheck__`? Whereas, `ABCMeta`'s definition of `__subclasshook__` *does* defer to the subclass's definition of `__subclasshook__`? — ely, Dec 16 '13 at 18:10
I edited your answer to add more explicit statement of your point. Please review it to make sure I understood you correctly and that my two additions are correct. If you agree, then I will accept the answer. Thanks for the help! — ely, Dec 16 '13 at 18:25

Why can `__subclasshook__` be monkeypatched onto the metaclass but `__instancecheck__` cannot?

1 Answers1

Why can `subclasshook` be monkeypatched onto the metaclass but `instancecheck` cannot?