Options for callback from python to c++

Question

Hello i've been trying to call a python user-defined callback from c++ using cython for a while. But it looks like it's impossible without changes on the c++ side or a static function buffer. So, is there only one option for binding a propper callback (ctypes with CFUNCTYPE)?

Cython 0.29.23

A.hpp:

typedef void (*Callback) ();

class A{
    Callback callback;
public:
    A(){
       this->callback = nullptr;
    }

    void set_callback(Callback callback){
        this->callback = callback;
    }

    void call_callback(){
        this->callback();
    }
};

A.pxd:

cdef extern from "A.hpp":
    ctypedef void (*Callback) ()
    cdef cppclass A:
        A() except +

        void set_callback(Callback callback)

        void call_callback()

B.pyx

from A cimport A, Callback

cdef class B:
    cdef A *c_self
    cdef object callback

    def __cinit__(self):
        self.c_self = new A()

    def __dealloc__(self):
        del self.c_self

    cdef void callback_func(self) with gil:
        print("I'm here")
        self.callback()

    def set_callback(self, callback):
        self.callback = callback
        self.c_self.set_callback(<Callback>self.callback_func)

    def call_callback(self):
        self.c_self.call_callback()


def print_():
    print("hello")

b = B()
b.set_callback(print)
b.call_callback()

Output:

I'm here
[segmentation fault]

Looks like ctypes: get the actual address of a c function is a good one work-around, but it uses ctypes. It scares me, but works:

B.pyx

from A cimport A, Callback
import ctypes
from libc.stdint cimport uintptr_t

cdef class B:
    cdef A *c_self
    cdef object callback

    def __cinit__(self):
        self.c_self = new A()

    def __dealloc__(self):
        del self.c_self

    def set_callback(self, callback):
        f = ctypes.CFUNCTYPE(None)(callback)
        self.callback = f
        cdef Callback c_callback = (<Callback*><uintptr_t>ctypes.addressof(f))[0]
        self.c_self.set_callback(c_callback)

    def call_callback(self):
        self.c_self.call_callback()


def hello():
    print("hello")

b = B()
b.set_callback(hello)
b.call_callback()

Answer 1

A function pointer does not have any space to store extra information. Therefore it is not possible to convert a Python callable to a function pointer in pure C. Similarly a cdef function of a cdef class must store the address of the instance to be usable and that is impossible too.

You have three options:

1. Use ctypes as in https://stackoverflow.com/a/34900829/4657412 (the bottom half of the answer shows how to do it). ctypes accomplishes this with runtime code generation (and thus only works on processors that it explicitly supports).

2. Use a std::function in C++ instead of a function pointer. These can store arbitrary information. You need to write an object wrapper (or re-use one from elsewhere) so it isn't completely pure Cython. See Pass a closure from Cython to C++. What you're trying to do is probably better covered by How to use a Cython cdef class member method in a native callback.

3. Use the class C scheme where the callback is of type

    void (CallbackT)(/* any args */, void* user_data)
   

   and is registered with:

    void register_callback(CallbackT func, void* user_data)
   

   In this case user_data would be the address of your B instances (and you'd need to make sure it was Py_INCREFed before setting the address and Py_DECREFed before unsetting the address). Cython callback with class method provides an example.