The deprecation of Python's PyEval_ReleaseLock has introduced a problem in our codebase: We want to terminate a Python interpreter from a C callback function using Py_EndInterpreter
So to do that, Python's Docs say that you must hold the GIL when calling this function:
void Py_EndInterpreter(PyThreadState *tstate)
Destroy the (sub-)interpreter represented by the given thread state. The given thread state must be the current thread state. See the discussion of thread states below. When the call returns, the current thread state is NULL. All thread states associated with this interpreter are destroyed. (The global interpreter lock must be held before calling this function and is still held when it returns.) Py_FinalizeEx() will destroy all sub-interpreters that haven’t been explicitly destroyed at that point.
Great! So we call PyEval_RestoreThread to restore our thread state to the thread we're about to terminate, and then call Py_EndInterpreter.
// Acquire the GIL
PyEval_RestoreThread(thread);
// Tear down the interpreter.
Py_EndInterpreter(thread);
// Now what? We still hold the GIL and we no longer have a valid thread state.
// Previously we did PyEval_ReleaseLock here, but that is now deprecated.
The documentation for PyEval_ReleaseLock says that we should either use PyEval_SaveThread or PyEval_ReleaseThread.
PyEval_ReleaseThread's documentation says that the input thread state must not be NULL. Okay, but we can't pass in the recently deleted thread state.
PyEval_SaveThread will hit a debug assertion if you try to call it after calling Py_EndInterpreter, so that's not an option either.
So, we've currently implemented a hack to get around this issue - we save the thread state of the thread that calls Py_InitializeEx in a global variable, and swap to it after calling Py_EndInterpreter.
// Acquire the GIL
PyEval_RestoreThread(thread);
// Tear down the interpreter.
Py_EndInterpreter(thread);
// Swap to the main thread state.
PyThreadState_Swap(g_init.thread_state_);
PyEval_SaveThread(); // Release the GIL. Probably.
What's the proper solution here? It seems that embedded Python is an afterthought for this API.
Similar question: PyEval_InitThreads in Python 3: How/when to call it? (the saga continues ad nauseam)
