0

I have a C script,

#include "main.h"


static const char *path_here = "./path/to/here";

static PyConfig py_config;

extern PyObject *fn_arg;


void *init_Python(void *) {

    pthread_cleanup_push(del_Python, NULL);

    /* Set-up the Python interpreter */
    PyConfig_InitPythonConfig(&py_config);
    setenv("PYTHONPATH", path_here, true);
    PyConfig_SetString(&py_config, &py_config.program_name, (wchar_t *)path_here);
    if (PyStatus_Exception(Py_InitializeFromConfig(&py_config)))
        del_Python(NULL);
    PyConfig_Clear(&py_config);

    /* Run the Python script */
    PyObject *fn_def = PyObject_GetAttrString(PyImport_ImportModule("test_module"), "test_function");
    if (!PyObject_CallOneArg(fn_def, fn_arg))
        del_Python(NULL);

    pthread_cleanup_pop(NULL);

    return NULL;
}


void del_Python(void *) {
    PyConfig_Clear(&py_config);

    /* ... cleanup in Python with e.g. PyRun_String() ... */

    Py_Finalize();

    exit(EXIT_SUCCESS);
}

which I want to pthread_create from another file. The Python module test_module contains a function, test_function, which functions as a context manager for a blocking I/O operation and is called from C using PyObject_CallOneArg.

How do I raise an exception in del_Python from C using e.g. PyRun_SimpleString to tell test_function to perform a cleanup?

I've tried to do something simple, i.e. change a variable, to test this out. However, PyRun_SimpleString("print(locals())"); reports garbage and even PyRun_String with PyEval_GetLocals() and PyEval_GetGlobals() fails silently when I attempt this (even though their respective dictionaries are populated properly with the relevant names). Ideally, I would want the Python interpreter to be "interrupted" and execute Python Bytecode compiled from C verbatim.

Furthermore, Py_Finalize() segfaults. Acquiring the GIL with PyGILState_Ensure() before calling Py_Finalize is a temporary fix but throws many Valgrind errors which all seem to relate to _PyEval_EvalFrameDefault.

I've read this, this, this, and this question but still can't figure it out. Did I simply skip over the solution or am I fundamentally misunderstand something? Can anybody please help me out?

Note: the "in 2023" in the title referred to most of the functions in the aforementioned questions being deprecated or removed entirely.

user169291
  • 21
  • 2
  • There's a lot here that doesn't make sense. Why would a *module* define `__call__`? How would that function as a context manager? Why are you trying to raise an exception at all, let alone in the middle of code you clearly want to keep executing after that point? – user2357112 Apr 24 '23 at 11:27
  • I have no control over the nomenclature of the Python code. I've changed ```__call__``` to ```test_function``` to avoid further confusion. I presume the context manager has been created with the ```context.lib.contextmanager``` decorator. I want to raise an exception to trigger cleanup which depends on the type of signal received rather than just kill the interpreter. – user169291 Apr 24 '23 at 11:37
  • Looking at the `pthread_cleanup_push(del_Python, NULL);`... are you trying to *cancel* a thread while it's executing Python code? That's going to make an unrecoverable mess of your Python interpreter state. CPython is not designed to handle that. – user2357112 Apr 24 '23 at 11:39
  • Well, I guess that's exactly what I want to do ;) I'd like to run some code, depending on the kind of signal that has been received, _before_ the thread kills itself. But I should have full control over the code executed before that point in time, right? What's the difference between that and e.g. intercepting a ```KeyboardInterrupt``` from Python directly? – user169291 Apr 24 '23 at 11:46
  • `KeyboardInterrupt` already has a nasty tendency to make an unrecoverable mess of whatever data structures a Python program is working with when it arrives, but at least it can't wreck the interpreter itself. It doesn't force-abort a thread's execution, and it can only arrive right before Python executes a new bytecode instruction, not in the middle of arbitrary C-level work. – user2357112 Apr 24 '23 at 12:12
  • But how does this force-abort a thread's execution? The thread is not exited until the end of the ```del_Python``` call. Is it not possible to somehow save the state of the interpreter/subinterpreter, "inject" new bytecode to raise whatever exception is necessary, have the relevant except clause clean it up, ```return``` to C, and shut down the interpreter gracefully? – user169291 Apr 24 '23 at 12:18
  • The process of unwinding the Python stack requires unwinding the *C* stack too. They're very closely tied together. You can't do that from a pthread cancellation handler. Plus, even if you could, you'd be interrupting at some unsafe point that was not designed to be interrupted that way. – user2357112 Apr 24 '23 at 12:31
  • Let us [continue this discussion in chat](https://chat.stackoverflow.com/rooms/253299/discussion-between-user169291-and-user2357112). – user169291 Apr 24 '23 at 12:52

0 Answers0