Have C extension class inherit from Python class

Question

I have a Python class AbstractFoo which uses @abc.abstractmethod to define 2 abstract methods.

For more performance the bulk of the work is implemented in C as a Python extension class CFoo, so what I wanted to do is to inherit that extension class from AbstractFoo.

However I haven't found a way to do this and went a different route: Implement it as a "normal" class in C and have a class ConcreteFoo(AbstractFoo, CFoo) which inherits from the abstract Python and the extension class.

However now the "protection" from the ABC didn't kick in: No error was thrown even when the methods were missing. I found a solution in https://stackoverflow.com/a/20440259/1930508 and added a new new method instead of PyType_GenericNew (ignore the ref counts of the empty_* parts for now):

static PyObject* CFoo_new(PyTypeObject* type, PyObject*, PyObject*)
{
    auto* empty_tuple(PyTuple_New(0));
    if (!empty_tuple)
        return nullptr;
    auto* empty_dict(PyDict_New());
    if (!empty_dict)
        return nullptr;
    return PyBaseObject_Type.tp_new(type, empty_tuple.get(), empty_dict.get());
}

But now the ABC check always triggers: TypeError: Can't instantiate abstract class ConcreteFoo with abstract methods ...
But checking dir(ConcreteFoo) shows the methods it complains about.

Is there a way to have an ABC-checked subclass of my AbstractFoo where the methods are implemented in C?

Edit: Some more code:

class AbstractFoo(abc.ABC):
    @abc.abstractmethod
    def register(self):
        pass
    # ...

#...

class ConcreteFoo(AbstractFoo, CFoo):
    def __init__(self, arg):
        AbstractFoo.__init__(self)
        CFoo.__init__(self, arg)

I think `__abstractmethods__` is set on the creation of the type by `ABCMeta`. Therefore, you need to ensure that this metaclass is called when you define the type `ConcreteFoo`, not `PyType_Type`. Unfortunately you don't show this bit of code... — DavidW, May 22 '20 at 18:34
I'm not sure what code you mean. The not shown code is pretty much the basic boilerplate code you use. I added the definition of `ConcreteFoo` and the important part of `AbstractFoo` above — Flamefire, May 25 '20 at 07:22
Ah - I misunderstood slightly and thought that you were declaring `ConcreteFoo` in C (e.g. with `PyObject_Call(PyType_Type,...`) That does make things clearer — DavidW, May 25 '20 at 07:26
`register` is a dangerous choice for an abstract method name, because it conflicts with [`abc`'s own use of the name `register`](https://docs.python.org/3/library/abc.html#abc.ABCMeta.register). — user2357112, May 25 '20 at 07:46

score 1 · Accepted Answer · answered May 25 '20 at 07:43

1

You have to make CFoo the first base class. This has little to do with the C-API and applies to a pure Python version too: if you define

class CFoo:
    def register(self):
        return "CFoo.register"

then

class ConcreteFoo(AbstractFoo, CFoo):

fails, but

class ConcreteFoo(CFoo, AbstractFoo):

works.

This makes sense in terms of the __mro__ and you can test this by making register not be an abstract method. You'll find that the method of the first base class in used in preference. When the first base class is AbstractFoo this is an abstract method that's being found first, so it fails.

answered May 25 '20 at 07:43

DavidW

29,336
6
55
86

This is correct and works, thanks. I find this counter-intuitive as I'd expect inheritance work from lft to right. For completeness: Do you happen to have a link to the official documentation? – Flamefire May 25 '20 at 07:50
Maybe https://www.python.org/download/releases/2.3/mro/ (although it obviously gets quite detailed quite quickly). Maybe the way to think about it is that all the methods in all the base classes still exist and it searches from left to right to find a suitable method (and not "methods get overridden by new base classes being added", which I think is what you're assuming) – DavidW May 25 '20 at 07:59

Have C extension class inherit from Python class

1 Answers1