Cython callback with class method

Question

I'm starting to use Cython to interface some C++ code with Python. Since I need to do some asynchronous data acquisition in C++, I was planning to use callbacks from my C++ code to Python. So far I've been able to successfully call Python functions from C++, however, I can't call Python class methods from C++. I've seen several questions asked on StackOverflow about it, but no one provides a simple answer to a seemingly simple thing to do. This is a working toy example.

myclass.h:

    #ifndef MYCLASS_H
    #define MYCLASS_H

    #define PY_SSIZE_T_CLEAN
    #include <Python.h>

    typedef void (*callbackfun)(PyObject* obj);

    class MyClass
    {
    public:
        void start(callbackfun user_func, PyObject* obj);
    };

    #endif // MYCLASS_H

myclass.cpp:

    #include "myclass.h"

    void MyClass::start(callbackfun user_func, PyObject* obj)
    {
        user_func(obj); 
    }

PyMyClass.pyx:

    # distutils: language = c++
    # cython: language_level=3

    cdef extern from "myclass.h":
        ctypedef void (*callbackfun)(object obj)

        cdef cppclass MyClass:
            MyClass()
            void start(callbackfun user_func, object obj)

    cdef void callback(self):
        print('Called back')

    cdef class PyMyClass:
        cdef MyClass *c_myClass_ptr

        def __cinit__(self):
            self.c_myClass_ptr = new MyClass()

        def __dealloc__(self):
            del self.c_myClass_ptr

        def start(self):
           self.c_myClass_ptr.start(callback, self)

main.py:

    from PyMyClass import PyMyClass

    def main():
        myClass = PyMyClass()
        myClass.start()

    if __name__ == "__main__":
        main()

Now, if I convert the callback function in PyMyClass.pyx to a method, as follows:

PyMyClass.pyx:

# distutils: language = c++
# cython: language_level=3

cdef extern from "myclass.h":
    ctypedef void (*callbackfun)(object obj)

    cdef cppclass MyClass:
        MyClass()
        void start(callbackfun user_func, object obj)

cdef class PyMyClass:
    cdef MyClass *c_myClass_ptr

    def __cinit__(self):
        self.c_myClass_ptr = new MyClass()

    def __dealloc__(self):
        del self.c_myClass_ptr

    cdef void callback(self):
        print('Called back')

    def start(self):
       self.c_myClass_ptr.start(self.callback, self)

Now the compiler complains with:

Compiling PyMyClass.pyx because it changed.
[1/1] Cythonizing PyMyClass.pyx

Error compiling Cython file:
------------------------------------------------------------
...

    cdef void callback(self):
        print('Called back')

    def start(self):
       self.c_myClass_ptr.start(self.callback, self)                                   ^
------------------------------------------------------------

PyMyClass.pyx:24:36: Cannot assign type 'void (PyMyClass)' to 'callbackfun'
Traceback (most recent call last):
  File "setup.py", line 22, in <module>
    setup(ext_modules = cythonize(ext_modules))
  File "/Users/andreac/anaconda3/lib/python3.7/site-packages/Cython/Build/Dependencies.py", line 1097, in cythonize
    cythonize_one(*args)
  File "/Users/andreac/anaconda3/lib/python3.7/site-packages/Cython/Build/Dependencies.py", line 1220, in cythonize_one
    raise CompileError(None, pyx_file)
Cython.Compiler.Errors.CompileError: PyMyClass.pyx

Is there a (simple) fix to this issue? (And yes, I understand that there's also a potential reference counting issue with using PyObject*)

score 1 · Accepted Answer · answered Sep 10 '19 at 18:21

You have a few problems, but they're relatively straightforward to fix:

self.callback is attempting to bind the function with the self pointer. I'm not 100% sure how this works in this case - it might end up creating a callable Python object, but either way it definitely doesn't match what you want to the callback function. Change it to &PyMyClass.callback. I've also added a & since Cython seems to require it to understand the situation.
&PyMyClass.callback still doesn't match the signature of callbackfun. callbackfun expects a generic Python object, while the first argument of PyMyClass.callback is specialized as a pointer to the underlying C struct for PyMyClass. This is easily solved with a cast, giving
```
self.c_myClass_ptr.start(<callbackfun>&PyMyClass.callback, self)
```
This casting to/from structs that are based on PyObject is a common idiom in using the Python C API, so is fine. Unfortunately casts can end up hiding errors, so be wary of that.
You have a potential reference counting disaster in the making: you pass a "borrowed reference" to self. You mention this in the question, but I'll just re-iterate in-case future readers missed it (as I did on a first skim). In this case it's fine - you immediately use self and then discard it. However, were you to save it in your C++ code then it could potentially be destructed leaving the C++ code with a valid pointer. This would really need a Py_INCREF when the callback is set up and a Py_DECREF when the callback is un-setup. Ensuring this happens can be hard to get right....

Cython callback with class method

1 Answers1

Linked