Why does super closure not use the new base class given by a metaclass?

Question

I'm toying with creating a class that dynamically has a base that is the class of an object passed to it on instantiation. E.g. pass it an int instance, and it subclasses int, etc. I'm trying to do this by employing a metaclass. This seems to break super. __class__ (a closure created if super exists) and the no-arguement form of super do not acknowledge the new base class.

With __class__ / super() broken, I'm not sure how to do what super() would have done. super(self.__class__, self) "works", but would break subclassing this class.

Why does __class__ not match self.__class__ in this case? I know they aren't equivalent concepts, but in here, I'd want them to have the same value. See the factory function alternative below. print(isinstance(self, Class)) also returns false. My guess is that because of the way super and __class__ are created when the code is parsed, it is initially set without considering the change in the metaclass, and the adoption of a new class created with type doesn't change the __class__ already set - though I'd have guessed it would. After all, you're supplying the new class, aren't you?

Can __class__ / super() be "fixed"? If it can't, can super still be somehow used cleanly?

I tried creating a class to do what I wanted super to do, with __getattribute__, but that's also pretty dicey.

See this runnable example with illustrative prints:

class Base:
    pass

class Meta(type):
    def __call__(mcls, obj, *args, **kwargs):
        bases = (obj.__class__,)

        __dict__ = dict(mcls.__dict__)
        if "__dict__" in __dict__:
            del __dict__['__dict__']
        if "__weakref__" in __dict__:
            del __dict__['__weakref__']

        cls = type(mcls.__name__, bases , __dict__)
        instance = cls.__new__(cls, *args, **kwargs)
        instance.__init__(*args, **kwargs)
        return instance

class Class(metaclass=Meta):
    def __init__(self, *args, **kwargs):
        print(__class__.__bases__)
        print(self.__class__.__bases__)
        print(isinstance(self, Class))
        print(isinstance(Class, Meta))
        print(isinstance(self, __class__))
        print(isinstance(self, self.__class__))

        super().__init__(*args, **kwargs)

In [1]: Class(Base())
(<class 'object'>,)
(<class '__main__.Base'>,)
False
True
False
True
---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
<ipython-input-55-6a5013764b7e> in <module>
----> 1 c = Class(Base())

<ipython-input-54-f1865febd768> in __call__(mcls, obj, *args, **kwargs)
     19         instance = cls.__new__(cls, *args, **kwargs)
     20         # Sets obj in instance for mutables
---> 21         instance.__init__(*args, **kwargs)
     22         return instance
     23

<ipython-input-54-f1865febd768> in __init__(self, *args, **kwargs)
     30         print(isinstance(self, self.__class__))
     31
---> 32         super().__init__(*args, **kwargs)
     33

TypeError: super(type, obj): obj must be an instance or subtype of type

---

Factory function

I'm aware that the above could be done simply with a factory function that looks like this:

class Base:
    def __init__(self, *args, **kwargs):
        print("Base init")

def get_instance(obj):
    class Class(type(obj)):
        def __init__(self, *args, **kwargs):
            print(__class__.__bases__)
            print(self.__class__.__bases__)
            super().__init__(*args, **kwargs)
    return Class()

which yields

In [1]: get_instance(Base())
Base init
(<class '__main__.Base'>,)
(<class '__main__.Base'>,)
Base init

In this example, super() finds the Base class because that's what it was defined with, and intuitively finds the Base's init dunder method. Rephrasing my question:

If I instead use a metaclass to dynamically subclass something, how can I get super() to find the dynamic base, just as it would if I had used a factory function? If it cannot (cleanly), are there reasonable alternatives?

Another phrasing

Finally, another way I think this question can be restated is:

How can normal subclassing be effectively emulated with a Metaclass?

Why?

Firstly, this behavior surprised me, and I'd like to understand it. In that regard this can be viewed as an academic question.

Beyond that, the question of how to dynamically sublcass isn't new, and there's clearly non-zero interest. But, the methods I've seen all seem lacking. super / __class__ seems problematic for metaclasses, and if that and everything else worked wonderfully (big if), there's still a concern about metaclass conflicts one might have to worry about in the general case. Using factory functions might seem simple, but they make it difficult to subclass the class you're creating. It's not straight forward to subclass the Class above in the factory function example. Using a metaclass, if you could get it to work, would still allow plainer access and subclassing.

Answer 1

The problem here is that your dynamic class does not inherit from itself (Class) at all - the implicit __class__ variable inside your __init__ points to the "hardcoded" "Class" version - but the self received when __init__ is called is an instance of the dynamic class, which does not have Class as its superclass. Thus, the arguments filled implicitly to super(): __class__ and self will mismatch (self is not an instance of the class defined in __class__ or of a subclass of it).

The reliable way to fix this is to allow proper inheritance, and forget copying the class __dict__ attributes around: let the inheritance machinery take care of calling the methods in the appropriate order.

By simply making the dynamic class inherit from your static class and the dynamic-base, all methods are in place, and self will be a proper instance from the baked-in __class__ from __init__.__class__ still points to the static Class, but the conditions for super to be called are fulfilled - and super does the right thing, by looking for the supermethods starting from the self parameter - the newly created dynamic subclass which inherits both from your static Class and the new Base, and calls the methods on Base as they are part of the __mro__ of the new class, in the correct place.

Ok - sounds complicated - but with some print statements added we can see the thing working:

class Base:
    def __init__(self):
        print("at base __init__")

class Meta(type):
    def __call__(cls, obj, *args, **kwargs):
        dynamic_ancestor = type(obj)
        bases = (cls, dynamic_ancestor,)

        new_cls = type(f"{cls.__name__}__{dynamic_ancestor.__name__}", bases , {})
        instance = new_cls.__new__(new_cls, *args, **kwargs)
        instance.__init__(*args, **kwargs)
        return instance

class Class(metaclass=Meta):
    def __init__(self, *args, **kwargs):
        print(__class__.__bases__)
        print(self.__class__.__bases__)
        print(isinstance(self, Class))
        print(isinstance(Class, Meta))
        print(isinstance(self, __class__))
        print(isinstance(self, self.__class__))
        print(self.__class__.__mro__, __class__.__mro__)
        super().__init__(*args, **kwargs)

Class(Base())

Outputs:

at base __init__
(<class 'object'>,)
(<class '__main__.Class'>, <class '__main__.Base'>)
True
True
True
True
(<class '__main__.Class__Base'>, <class '__main__.Class'>, <class '__main__.Base'>, <class 'object'>) (<class '__main__.Class'>, <class 'object'>)
at base __init__