How can I override a special method defined in a metaclass with a custom classmethod?

Question

As an example, consider the following:

class FooMeta(type):
    def __len__(cls):
        return 9000


class GoodBar(metaclass=FooMeta):
    def __len__(self):
        return 9001


class BadBar(metaclass=FooMeta):
    @classmethod
    def __len__(cls):
        return 9002

len(GoodBar) -> 9000
len(GoodBar()) -> 9001
GoodBar.__len__() -> TypeError (missing 1 required positional argument)
GoodBar().__len__() -> 9001
len(BadBar) -> 9000 (!!!)
len(BadBar()) -> 9002
BadBar.__len__() -> 9002
BadBar().__len__() -> 9002

The issue being with len(BadBar) returning 9000 instead of 9002 which is the intended behaviour.

This behaviour is (somewhat) documented in Python Data Model - Special Method Lookup, but it doesn't mention anything about classmethods, and I don't really understand the interaction with the @classmethod decorator.

Aside from the obvious metaclass solution (ie, replace/extend FooMeta) is there a way to override or extend the metaclass function so that len(BadBar) -> 9002?

Edit:

To clarify, in my specific use case I can't edit the metaclass, and I don't want to subclass it and/or make my own metaclass, unless it is the only possible way of doing this.

Answer 1

As you already mentioned, len() is going to find __len__ on the object's type not the object itself. So len(BadBar) is called like type(BadBar).__len__(BadBar), FooMeta's __len__ method is getting the instance which is BadBar here as the first parameter.

class FooMeta(type):
    def __len__(cls):
        print(cls is BadBar)
        return 9000


class BadBar(metaclass=FooMeta):
    @classmethod
    def __len__(cls):
        return 9002


print(len(BadBar)) # True - 9000

This is how it works, if you do anything with metaclass's __len__, other classes with that metaclass will be affected. You can not generalize it.

You can have another specific metaclass for BadBar and call the cls.__len__ inside the __len__ of the metaclass if it has one, otherwise return the default value.

class BadBar_meta(type):
    def __len__(cls):
        if hasattr(cls, "__len__"):
            return cls.__len__()
        return 9000


class BadBar(metaclass=BadBar_meta):
    @classmethod
    def __len__(cls):
        return 9002


print(len(BadBar))  # 9002

Or if you just want to override only classmethods of the classes, you can check to see if the cls class has already defined __len__ as a classmethod, if so call that instead.

class FooMeta(type):
    def __len__(cls):
        m = cls.__len__
        if hasattr(m, "__self__"):
            return cls.__len__()

        return 9000


class GoodBar(metaclass=FooMeta):
    def __len__(self):
        return 9001


class BadBar(metaclass=FooMeta):
    @classmethod
    def __len__(cls):
        return 9002


print(len(GoodBar)) # 9000
print(len(BadBar))  # 9002

Here are more robust approaches to find classmethods.

After your edit:

If you don't have access to the metaclass, then you can decorate and monkey patch the metaclass:

class FooMeta(type):
    def __len__(cls):
        return 9000


class GoodBar(metaclass=FooMeta):
    def __len__(self):
        return 9001


class BadBar(metaclass=FooMeta):
    @classmethod
    def __len__(cls):
        return 9002


def decorator(mcls):
    org_len = mcls.__len__

    def custom_len(cls):
        # check too see if it is a classmethod
        if hasattr(cls.__len__, "__self__"):
            return cls.__len__()
        return org_len(cls)

    FooMeta.__len__ = custom_len
    return mcls


FooMeta = decorator(FooMeta)

print(len(GoodBar))  # 9000
print(len(BadBar))   # 9002

Answer 2

The __len__ defined in the class will always be ignored when using len(...) for the class itself: when executing its operators, and methods like "hash", "iter", "len" can be roughly said to have "operator status", Python always retrieve the corresponding method from the class of the target, by directly acessing the memory structure of the class. These dunder methods have "physical" slot in the memory layout for the class: if the method exists in the class of your instance (and in this case, the "instances" are the classes "GoodBar" and "BadBar", instances of "FooMeta"), or one of its superclasses, it is called - otherwise the operator fails.

So, this is the reasoning that applies on len(GoodBar()): it will call the __len__ defined in GoodBar()'s class, and len(GoodBar) and len(BadBar) will call the __len__ defined in their class, FooMeta

I don't really understand the interaction with the @classmethod decorator.

The "classmethod" decorator creates a special descriptor out of the decorated function, so that when it is retrieved, via "getattr" from the class it is bound too, Python creates a "partial" object with the "cls" argument already in place. Just as retrieving an ordinary method from an instance creates an object with "self" pre-bound:

Both things are carried through the "descriptor" protocol - which means, both an ordinary method and a classmethod are retrieved by calling its __get__ method. This method takes 3 parameters: "self", the descriptor itself, "instance", the instance its bound to, and "owner": the class it is ound to. The thing is that for ordinary methods (functions), when the second (instance) parameter to __get__ is None, the function itself is returned. @classmethod wraps a function with an object with a different __get__: one that returns the equivalent to partial(method, cls), regardless of the second parameter to __get__.

In other words, this simple pure Python code replicates the working of the classmethod decorator:

class myclassmethod:
    def __init__(self, meth):
         self.meth = meth
    def __get__(self, instance, owner):
         return lambda *args, **kwargs: self.meth(owner, *args, **kwargs)

That is why you see the same behavior when calling a classmethod explicitly with klass.__get__() and klass().__get__(): the instance is ignored.

TL;DR: len(klass) will always go through the metaclass slot, and klass.__len__() will retrieve __len__ via the getattr mechanism, and then bind the classmethod properly before calling it.

Aside from the obvious metaclass solution (ie, replace/extend FooMeta) is there a way to override or extend the metaclass function so that len(BadBar) -> 9002?

(...) To clarify, in my specific use case I can't edit the metaclass, and I don't want to subclass it and/or make my own metaclass, unless it is the only possible way of doing this.

There is no other way. len(BadBar) will always go through the metaclass __len__.

Extending the metaclass might not be all that painful, though. It can be done with a simple call to type passing the new __len__ method:

In [13]: class BadBar(metaclass=type("", (FooMeta,), {"__len__": lambda cls:9002})):
    ...:     pass
    

In [14]: len(BadBar)
Out[14]: 9002

Only if BadBar will later be combined in multiple inheritance with another class hierarchy with a different custom metaclass you will have to worry. Even if there are other classes that have FooMeta as metaclass, the snippet above will work: the dynamically created metaclass will be the metaclass for the new subclass, as the "most derived subclass".

If however, there is a hierarchy of subclasses and they have differing metaclasses, even if created by this method, you will have to combine both metaclasses in a common subclass_of_the_metaclasses before creating the new "ordinary" subclass.

If that is the case, note that you can have one single paramtrizable metaclass, extending your original one (can't dodge that, though)

class SubMeta(FooMeta):
    def __new__(mcls, name, bases, ns, *,class_len):
         cls = super().__new__(mcls, name, bases, ns)
         cls._class_len = class_len
         return cls

    def __len__(cls):
        return cls._class_len if hasattr(cls, "_class_len") else super().__len__()

And:

In [19]: class Foo2(metaclass=SubMeta, class_len=9002): pass

In [20]: len(Foo2)
Out[20]: 9002