Python C extension: Segfault when checking if Python callback is valid

Question

I am trying to register a Python callback to my C library. When checking if the callback is a valid Python callback with PyCallable_Check() it segfaults...

Here is the simple code to duplicate the issue:

#include <Python.h>
#include <stdio.h>

typedef void (*event_handler_t)(const void* uuid, const uint8_t* data, size_t data_length, void* user_data);

PyObject * m_handler;
void *m_user_data;

void register_notification(void* connection, PyObject * event_handler, void* user_data) {
    if (!PyCallable_Check(event_handler)) {
        printf("parameter must be callable\n");
        return;
    }

    m_handler = event_handler;
    m_user_data = user_data;
}

To build it:

gcc -fPIC -shared `pkg-config --cflags python-3.6` my_module.c `pkg-config --libs python-3.6` -o minimal_check.so`

And here is the python script to trigger the issue:

from ctypes import *

mymodule = CDLL("minimal_check.so")

event_handler_type = CFUNCTYPE(None, c_void_p, c_void_p, c_size_t, c_void_p)

register_notification = mymodule.register_notification
register_notification.argtypes = [c_void_p, event_handler_type, c_void_p]

def my_callback(uuid_ptr, data, data_len, user_data):
    pass

register_notification(None, event_handler_type(my_callback), None)

Looking at the coredump, the crash seems to definitely be in PyCallable_Check:

#0  0x0000000000531a0d in PyCallable_Check () at ../Objects/object.c:1307
#1  0x00007febcd24e72a in register_notification () from /tmp/minimal_check/minimal_check.so
#2  0x00007febcd455dae in ffi_call_unix64 () from /usr/lib/x86_64-linux-gnu/libffi.so.6
#3  0x00007febcd45571f in ffi_call () from /usr/lib/x86_64-linux-gnu/libffi.so.6
#4  0x00007febcd6a1c64 in _call_function_pointer (argcount=3, resmem=0x7ffe670393c0, restype=<optimized out>, atypes=0x7ffe67039380, avalues=0x7ffe670393a0, pProc=0x7febcd24e70a <register_notification>, 
    flags=4353) at ./Modules/_ctypes/callproc.c:831
#5  _ctypes_callproc () at ./Modules/_ctypes/callproc.c:1195
#6  0x00007febcd6a2404 in PyCFuncPtr_call () at ./Modules/_ctypes/_ctypes.c:3970
#7  0x000000000057ec0c in _PyObject_FastCallDict (kwargs=<optimized out>, nargs=<optimized out>, args=<optimized out>, func=<_FuncPtr(__name__='register_notification') at remote 0x7febcf541750>)
    at ../Objects/tupleobject.c:131
#8  _PyObject_FastCallKeywords () at ../Objects/abstract.c:2496
#9  0x00000000004f88ba in call_function () at ../Python/ceval.c:4875
#10 0x00000000004f98c7 in _PyEval_EvalFrameDefault () at ../Python/ceval.c:3335
#11 0x00000000004f6128 in PyEval_EvalFrameEx (throwflag=0, f=Frame 0x140ebb8, for file minimal_check.py, line 15, in <module> ()) at ../Python/ceval.c:4166
#12 _PyEval_EvalCodeWithName.lto_priv.1581 () at ../Python/ceval.c:4166
#13 0x00000000004f9023 in PyEval_EvalCodeEx (closure=0x0, kwdefs=0x0, defcount=0, defs=0x0, kwcount=0, kws=0x0, argcount=0, args=0x0, locals=<optimized out>, globals=<optimized out>, _co=<optimized out>)
    at ../Python/ceval.c:4187
#14 PyEval_EvalCode (co=<optimized out>, globals=<optimized out>, locals=<optimized out>) at ../Python/ceval.c:731
#15 0x00000000006415b2 in run_mod () at ../Python/pythonrun.c:1025
#16 0x000000000064166a in PyRun_FileExFlags () at ../Python/pythonrun.c:978
#17 0x0000000000643730 in PyRun_SimpleFileExFlags () at ../Python/pythonrun.c:419
#18 0x000000000062b26e in run_file (p_cf=0x7ffe670399ec, filename=<optimized out>, fp=<optimized out>) at ../Modules/main.c:340
#19 Py_Main () at ../Modules/main.c:810
#20 0x00000000004b4cb0 in main (argc=2, argv=0x7ffe67039be8) at ../Programs/python.c:69

Good point, I fixed it in another version but forgot in this one. Fixing `c_size_t` did not change anything :-( — OlivierM, Jul 07 '19 at 12:55
I thought in case the callback does not return any value (`void`-type function) we could use `None`. Again, changing `None` to `c_int` and returning `0` in my callback did not fix the SegFault. — OlivierM, Jul 07 '19 at 12:58
related: https://stackoverflow.com/questions/33484591/callbacks-with-ctypes-how-to-call-a-python-function-from-c — Jean-François Fabre, Jul 07 '19 at 13:12
@Jean-FrançoisFabre I follow the similar steps the other SO thread mentions. I have not the impression I missed something. I have just tried with the decorator rather than the 'casting' to the function type as mentioned in the second answer of the SO thread - but I have still the issue. — OlivierM, Jul 07 '19 at 13:22
Looking at the code of `PyCallable_Check`: https://github.com/python/cpython/blob/3.6/Objects/object.c#L1303 The code seems to be straightforward. I edited my C function to print the PyObject using `PyObject_Str()` and it also crashes in this function. So the issue seems that `event_handler` is not a valid `PyObject`. — OlivierM, Jul 07 '19 at 13:34
this method is basically `return x->ob_type->tp_call != NULL;` after null pointer check on x BTW... — Jean-François Fabre, Jul 07 '19 at 13:38
Yep, that's why I meant my PyObject does not seem to be valid. `x->ob_type` is likely to not be a valid pointer. — OlivierM, Jul 07 '19 at 13:40
You're **totally** confusing 2 concepts here. Ctypes is meant for functions that know **nothing** about the python. You're creating a C function pointer in the python code but C code expects to get a python method!! — Antti Haapala -- Слава Україні, Jul 07 '19 at 13:44
And most importantly this is **not** at all how python extensions work. — Antti Haapala -- Слава Україні, Jul 07 '19 at 13:49

OlivierM · Accepted Answer · 2019-07-07T14:35:03.430

Based on @AnttiHaapala comments, I managed to get something working. So I:

removed CFUNCTYPE as it is a Python Object Callback that is passed to
passed directly the python callback to register_notification()
changed the callback type to py_object

And it seems to work. For the sake of the demonstration and in case it could help future fellows who would have the same issue I added code to demonstrate the callback.

Here is the new code:

#include <Python.h>
#include <stdio.h>

typedef void (*event_handler_t)(void* user_data);

PyObject *m_handler;
PyObject *m_user_data;

void register_notification(void* connection, PyObject *event_handler, PyObject *user_data) {
    if (!PyCallable_Check(event_handler)) {
        printf("parameter must be callable\n");
        return;
    }

    m_handler = event_handler;
    m_user_data = user_data;

    PyGILState_STATE d_gstate;
    d_gstate = PyGILState_Ensure();

    PyObject *arglist = Py_BuildValue("(O)", user_data);
    PyObject *result = PyEval_CallObject(event_handler, arglist);

    PyGILState_Release(d_gstate);

    if (result != NULL) {
        Py_DECREF(result);
    }
}

And the python code:

from ctypes import *

mymodule = CDLL("minimal_check.so")

register_notification = mymodule.register_notification
register_notification.argtypes = [c_void_p, py_object, py_object]

def my_callback(user_data):
    print("From callback: %s" % user_data)

register_notification(c_void_p(None), my_callback, "Hello World")

Python C extension: Segfault when checking if Python callback is valid

1 Answers1