Python callbacks on different threads & the GIL

Question

I am wrapping C functions that belong to a multi threaded C framework in a python module. In this framework there are callbacks that are executed when certain events are triggered. However, the callbacks are not always executed from the same thread.

The callbacks that are related to my problem are defined in python ctypes. The problem with normal ctypes callbacks is that they try to acquire the GIL using the python C-API. This aquisition fails when it is ran from a thread that did not call Py_initialize().

Because of the framework I cannot change the way the callbacks are executed, the only thing I can do is change the callback functions themselves.

To enable control over the way the GIL is acquired I have made a C callback function that wraps the python callback function:

void* callback(void*){
    if (PyGILState_GetThisThreadState()) {
        PyGILState_STATE gstate;
        gstate = PyGILState_Ensure();
        PyRun_SimpleString(...);
        PyGILState_Release(gstate);
    } else {
        PyRun_SimpleString(...);
    }
}

If the PyGILState_GetThisThreadState() functions returns a non NULL value it means the callback is executed from the same thread as Py_initialize() was called. This means that the PyGILState API can be used. This works properly.

The else part is executed when the callback is ran from an other thread. It will be executed properly if it is the only python call on the stack of that thread, but when there is a unfinished python call in the stack it will result in a Segmentation fault:

#0  0x0000003c46ce6cdb in PyImport_GetModuleDict () from /usr/lib64/libpython2.6.so.1.0
#1  0x0000003c46ce6d2b in PyImport_AddModule () from /usr/lib64/libpython2.6.so.1.0`
#2  0x0000003c46cf2fe8 in PyRun_SimpleStringFlags () from /usr/lib64/libpython2.6.so.1.0
...
<framework calls>
...
#19 0x0000003c46cf1daa in PyRun_StringFlags () from /usr/lib64/libpython2.6.so.1.0
#20 0x0000003c46cf3010 in PyRun_SimpleStringFlags () from /usr/lib64/libpython2.6.so.1.0
#21 0x00007fc72c78dbb6 in execString () from ...

I have tried using the PyGILState API for the else part as well, but it results in the opposite situation: The callback succeeds for successive python calls but results in a deadlock when it is the first python call in the stack:

#0  0x0000003c8de0da00 in sem_wait () from /lib64/libpthread.so.0
#1  0x0000003c46cfd428 in PyThread_acquire_lock () from /usr/lib64/libpython2.6.so.1.0
#2  0x0000003c46cd7784 in PyEval_RestoreThread () from /usr/lib64/libpython2.6.so.1.0
#3  0x0000003c46cf0f58 in PyGILState_Ensure () from /usr/lib64/libpython2.6.so.1.0

If i get the GIL state in the same manner as the if(PyGILState_GetThisThreadState()){..} part, with PyGILState_Ensure and PyGILState_Release, it will work correctly while there is a earlier python call in the stack, but not when there is no python call in the stack.

Is there a way to determine if the callback is about to make the first python or a successive call? Or perhaps a way to work around this threading problem?

score 1 · Answer 1 · edited May 23 '17 at 12:30

Apparently I can use _PyThreadState_Current, I am still not 100% sure why this works, but it does. It seems it should be currentThread instead of !currentThread.

void callback(){
    PyThreadState * currentThread = _PyThreadState_Current;
    if (PyGILState_GetThisThreadState() || !currentThread){
        PyGILState_STATE gstate;
        gstate = PyGILState_Ensure();
        PyRun_SimpleString(....);
        PyGILState_Release(gstate);
    }else{
        PyRun_SimpleString(....);
    }

Thanks to this post for pointing me in the right direction.

Python callbacks on different threads & the GIL

1 Answers1