Handling embedded Python interpreter calls with GIL and multi-threading

Question

Constellation / Context:

A C++ Executable (1) which dynamically links a C++ shared library emb.so (2) which in turn is running an embedded python interpreter (3) that calls custom python functions (4).

Embedding the Python interpreter (3) is happening by using pybind11. A call to a Python function from C++ can be simplified as:

py::module::import("test").attr("my_func")();

The executable (1) has a main loop in which it can do some other work, but it will call the python function at regular intervals.

Observation:

Variant 1: If I block inside the python function, the python code executes smoothly and quickly, but the main executable loop is obviously blocked
Variant 2: If I create a python thread inside the python function to return from the function immediately, the main executable is running, but the python code is running extremely slow (I can watch the iterations of a for-loop with a print one by one)

Question:

Why is Variant 2 so slow and how can I fix it?

My guess is that this has something to do with the GIL, and I tried to release the GIL inside the wrapper emb.so before returning to the main loop, but I wasn't able to do this without a segfault.

Any ideas?

score 2 · Answer 1 · answered May 08 '20 at 09:48

It turned out that this is very much related to the following question:

Embedding python in multithreaded C application

(see answer https://stackoverflow.com/a/21365656/12490068)

I solved the issue by explicitely releasing the GIL after calling embedded Python code like this:

state = PyGILState_Ensure();
// Call Python/C API functions...    
PyGILState_Release(state);

If you are doing this in a function or other C++ scope and you are creating python objects, you have to make sure that the python object's desctructor is not called after releasing the GIL. So don't do:

void my_func() {
    gil_state = PyGILState_Ensure();
    py::int_ ret = pymodule->attr("GiveMeAnInt")();
    PyGILState_Release(gil_state);
    return ret.cast<int>();
}

but instead do

void my_func() {
    int ret_value;
    gil_state = PyGILState_Ensure();
    {
        py::int_ ret = pymodule->attr("GiveMeAnInt")();
        ret_value = ret.cast<int>();
    }
    PyGILState_Release(gil_state);
    return ret_value;
}

Handling embedded Python interpreter calls with GIL and multi-threading

1 Answers1