Python: Subclass whole hierarchy

Question

My solution is at the bottom of the question, based on MisterMiyagi's example

---

I wasn't sure how best to phrase the title. My idea is the following. I have an abstract base class with some implementations. Some of these implementations refer to eachother as part of their logic, simplified as follows:

import abc


# the abstract class
class X(abc.ABC):
    @abc.abstractmethod
    def f(self):
        raise NotImplementedError()

# first implementation
class X1(X):
    def f(self):
        return 'X1'

# second implementation, using instance of first implementation
class X2(X):
    def __init__(self):
        self.x1 = X1()

    def f(self):
        return self.x1.f() + 'X2'

# demonstration
x = X2()
print(x.f())  # X1X2
print(x.x1.f())  # X1

Now I want to use these classes somewhere, let's say in another module. However I want to add some extra functionality (for example a function g) to all classes in the hierarchy. I could do that by adding it to the base class X, but I want to keep the functionality defined separately. For example I might want to define the new functionality like this:

class Y(X):
    def g(self):
        return self.f() + 'Y1'

This creates another base class with the new functionality, but of course doesn't add it to the existing implementations X1 and X2. I'd have to use diamond inheritance to get that:

class Y1(X1, Y):
    pass

class Y2(X2, Y):
    pass

# now I can do this:
y = Y2()
print(y.g())  # X1X2Y1

The above works correctly, but there is still a problem. In X2.__init__, an instance of X1 is created. For my idea to work this would have to become Y1 in Y2.__init__. But this is of course not the case:

print(y.x1.g())  # AttributeError: 'X1' object has no attribute 'g'

I think what I might be looking for is a way to turn X into an abstract metaclass, such that its implementations require a 'base' parameter to become classes, which can then be instantiated. This parameter is then used within the class to instantiate other implementations with the correct base.

Creating an instance with the new functionality in the base class would then look something like this:

class Y:
    def g(self):
        return self.f() + 'Y1'

X2(Y)()

Which would result in an object equivalent to an instance of the following class:

class X2_with_Y:
    def __init__(self):
        self.x1 = X1(Y)()

    def f(self):
        return self.x1.f() + 'X2'

    def g(self):
        return self.f() + 'Y1'

However I don't know how to create a metaclass that would do this. I'd like to hear whether a metaclass is the right idea and, if so, how to do it.

---

Solution

Using MisterMiyagi's example I was able to get something that I think will work. The behaviour is close to the metaclass idea that I had.

import abc


class X(abc.ABC):
    base = object  # default base class

    @classmethod
    def __class_getitem__(cls, base):
        if cls.base == base:
            # makes normal use of class possible
            return cls
        else:
            # construct a new type using the given base class and also remember the attribute for future instantiations
            name = f'{cls.__name__}[{base.__name__}]'
            return type(name, (base, cls), {'base': base})


    @abc.abstractmethod
    def f(self):
        raise NotImplementedError()


class X1(X):
    def f(self):
        return 'X1'


class X2(X):
    def __init__(self):
        # use the attribute here to get the right base for the other subclass
        self.x1 = X1[self.base]()

    def f(self):
        return self.x1.f() + 'X2'


# just the wanted new functionality
class Y(X):
    def g(self):
        return self.f() + 'Y1'

Usage is like this:

# demonstration
y = X2[Y]()
print(y.g())  # X1X2Y1
print(y.x1.g())  # X1Y1
print(type(y))  # <class 'abc.X2[Y]'>
# little peeve here: why is it not '__main__.X2[Y]'?

# the existing functionality also still works
x = X2()
print(x.f())  # X1X2
print(x.x1.f())  # X1
print(type(x))  # <class '__main__.X2'>

Answer 1

Since you are looking at a way to customise your classes, the easiest approach is to do just that:

class X2(X):
    component_type = X1

    def __init__(self):
        self.x1 = self.component_type()

class Y2(X2, Y):
    component_type = Y1

Since component_type is a class attribute, it allows to specialise different variants (read: subclasses) of the same class.

---

Note that you can of course use other code to construct such classes. Classmethods can be used to create new, derived classes.

Say that for example your classes are capable of picking the correct subclasses from their hierarchy.

class X2(X):
    hierarchy = X

    @classmethod
    def specialise(cls, hierarchy_base):
        class Foo(cls, hierarchy_base):
            hierarchy = hierarchy_base
        return Foo

Y2 = X2.specialise(Y)

Since Python3.7, it is possible to use __class_getitem__ to write the above as Y2 = X2[Y], similar to how Tuple can be specialised to Tuple[int].

class X2(X):
    hierarchy = X

    def __class_getitem__(cls, hierarchy_base):
        class Foo(cls, hierarchy_base):
            hierarchy = hierarchy_base
        return Foo

Y2 = X2[Y]

---

Class attributes often serve the functionality of metaclass fields, since they express precisely that. In theory, setting a class attribute is equivalent to adding a metaclass field, then setting it as an attribute for each class. What is exploited here is that Python allows instances to have attributes without classes defining their fields. One can think of class attributes as duck-typing metaclasses.

Answer 2

You problem is one that arrives quite often, namely that you want to change the behavior of an existing class. The one aspect you can achieve by inheriting it and adding your new behavior. All instances you then create of this subclass have the new behavior.

But you also want that someone else (in this case it's X2) who creates further instances of this class now instead creates instances of your own subclass with the added behavior instead.

This can be considered to be meddling in the affairs of someone else. I mean, if the class X2 wants to create an instance of X1, who are you (a mere user of X2!) to tell it that it shall instead create something else?? Maybe it doesn't work properly with something which is not of type X1!

But—of course. Been there. Done that. I know that sometimes this need arises.

The straight way to achieve this is to make the class X2 cooperate. That means, instead of creating an instance of the class X1 it could instead create an instance of a class passed as parameter:

class X2(X):
    def __init__(self, x1_class=X1):
        self.x1 = x1_class()

This could also be nicely embedded using method overriding instead of parameter passing:

class X2(X):
    @classmethod
    def my_x1(cls):
         return X1

    def __init__(self):
        self.x1 = self.my_x1()

and then in the other module:

class Y2(X2, Y):
    @classmethod
    def my_x1(cls):
        return Y1

But all this just works if you can change the X2, and in some cases you cannot do this (because the module of the X is third-party provided or even a builtin library, so effectively read-only).

In these cases you can consider monkey-patching:

def patched_init(self):
    self.x1 = Y1()

X1.__init__ = patched_init

Similar solutions can be approached using mocks as known from unit test modules. but all these have in common that they are applied to intimate details of the current implementations of the used classes. As soon as these change, the code breaks.

So if you can, it's way better to prepare the base classes (X2) to your project and make it more flexible for your use case.