How to set python function as callback for c++ using pybind11?

Question

typedef bool (*ftype_callback)(ClientInterface* client, const Member* member ,int member_num);

struct Member{
    char x[64];
    int y;
};

class ClientInterface {
public: 
    virtual int calc()=0;
    virtual bool join()=0;
    virtual bool set_callback(ftype_callback on_member_join)=0;
};

It is from SDK which I can call the client from dynamic library in c++ codes.

bool cb(ClientInterface* client, const Member* member ,int member_num) {
    // do something
}
cli->set_callback(cb);
cli->join();

I want to port it to python bindings use pybind11. How do I set_callback in python?

I have seen the doc and try:

PYBIND11_MODULE(xxx, m) {
    m.def("set_callback", [](xxx &self, py::function cb ){
        self.set_callback(cb);
    });
}

The code just failed to compile.

My question, how do I convert the py::function to ftype_callback or there is other way to make it?

Where do you get the `cb`? Do you define it in your C++ code or in Python code? — Nimrod, Jan 06 '22 at 07:45
@Nimrod the example codes is defined in C++. I want to define `callback function` in Python and `set_callback` in Python so the SDK will call the `callback function in Python code. — HALF9000, Jan 06 '22 at 07:48
Have you ever seen the [doc](https://pybind11.readthedocs.io/en/stable/advanced/cast/functional.html)? Is it helpful? — Nimrod, Jan 06 '22 at 07:57
Not sure what you want is directly possible. The [list of builtin conversions](https://pybind11.readthedocs.io/en/stable/advanced/cast/overview.html) does not include function pointers. It includes `std::function`, which is much more flexible. If you can change the C++ code, consider changing `ftype_callback` to `using ftype_callback = std::function;` and use `ftype_callback&` in the lambda argument, similarly to the `func_arg` example in the documentation you linked to. — Roee Shenberg, Jan 06 '22 at 09:49

Shahriar · Accepted Answer · 2022-01-17T15:18:30.240

You need a little C++ to get things going. I'm going to use a simpler structure to make the answer more readable. In your binding code:

#include <pybind11/pybind11.h>

#include <functional>
#include <string>

namespace py = pybind11;

struct Foo
{
    int i;
    float f;
    std::string s;
};

struct Bar
{
    std::function<bool(const Foo &foo)> python_handler;
    std::function<bool(const Foo *foo)> cxx_handler;

    Bar()
    {
        cxx_handler = [this](const Foo *foo) { return python_handler(*foo); };
    }
};

PYBIND11_MODULE(example, m)
{
    py::class_<Foo>(m, "Foo")  //
        .def_readwrite("i", &Foo::i)
        .def_readwrite("f", &Foo::f)
        .def_readwrite("s", &Foo::i);

    py::class_<Bar>(m, "Bar")  //
        .def_readwrite("handler", &Bar::python_handler);
}

Here, Foo is the object that is passed to the callback, and Bar is the object that needs its callback function set. Since you use pointers, I have wrapped the python_handler function with cxx_handler that is meant to be used in C++, and converted the pointer to reference.

To be complete, I'll give a possible example of usage of the module here:

import module.example as impl

class Bar:
    def __init__(self):
        self.bar = impl.Bar()
        self.bar.handler = self.handler
        
    def handler(self, foo):
        print(foo)
        return True

I have used this structure successfully in one of my projects. I don't know how you want to proceed, but perhaps if you don't want to change your original structure you can write wrapper classes upon them that use the given structure.

Update:

I thought that you controlled the structure when I wrote the answer above (I'll keep it for anyone who needs it). If you have a single cli instance, you can do something like:

using Handler = std::function<bool(std::string, int, int)>;

Handler handler;

bool cb(ClientInterface *client, const Member *member, int member_num)
{
    return handler(std::string(member->x), member->y, member_num);
}

// We have created cli somehow

// cli->set_callback(cb);

// cli->join();

PYBIND11_MODULE(example, m)
{
    m.def("set_callback", [](Handler h) { handler = h; });
}

If you have multiple ClientInterface instances, you can map ClientInterface pointers to handlers and call the appropriate handler in the cb function based on given ClientInterface pointer.

Note: I haven't tested the above with a python script but it should work.

Another Update

If you want to handle multiple instances, the code can roughly look like this:

using Handler = std::function<bool(std::string, int, int)>;

std::map<ClientInterface *, handler> map;

bool cb(ClientInterface *client, const Member *member, int member_num)
{
    // Check if <client> instance exists in map
    return map[client](std::string(member->x), member->y, member_num);
}

PYBIND11_MODULE(example, m)
{
    m.def("set_callback", [](int clientid, Handler h) 
    {
        // Somehow map <clientid> to <client> pointer
        map[client] = h;
    });
}

Note that this isn't a runnable code and you need to complete it.

Thanks for the answer. My question is that the `set_callback` is defined in the SDK to use `function pointer`. I am not allowed to change it. I can not figure out how should I use your pattern to tackle it. — HALF9000, Jan 14 '22 at 03:36
May I ask how to map multiple instances to different handler? I implement the callback function use this pattern. However, it seems the `set_callback` will set the `callback` among all the threads in python. Multiprocessing is fine. — HALF9000, Jan 17 '22 at 12:24
I edited the answer to include mapping demonstration. To handle multiple threads, maybe you can open one client per thread or something like that. It all depends on what you're trying to achieve. Hopefully the update will solve your problem. — Shahriar, Jan 17 '22 at 15:23

score 0 · Answer 2 · answered Sep 02 '22 at 02:04

you can get data using python types in pybind11

make sure you have #include <pybind11/functional.h>

// c++
using PyCallback = std::function<void(pybind11::bytearray)>;

class Haha
{
public:
    void setCallback(PyCallback& pyfn) {
        m_pyfn = pyfn;
    }
    void onDataAvaiable(char* buf, int len) {
        m_pyfn(pybind11::bytearray(buf, len));
    }
private:
    PyCallback m_pyfn;
};

PYBIND11_MODULE(haha, m) {
    pybind11::class_<Haha>(m, "Haha")
        .def("setCallback", &Haha::setCallback);
}

// python
def fn(data):
    print(data)

hahaInstance = m.Haha()
hahaInstance .setCallback(fn)
while True:
    // block to make sure hahaInstance is always running, then callback will print data

I think this answer could be improved by using similar identifiers and style conventions as the asker used in their code. — starball, Sep 05 '22 at 18:36

How to set python function as callback for c++ using pybind11?

2 Answers2

Update:

Another Update

Linked