How do I inject a catchable exception in Python from C with C Python API?

Question

I am executing the following Python function from C:

Python:

def runLoop(self):
    try:
        while True:
            print("working")
            time.sleep(1)  # Delay for 1 second.
            self.incrementcounter()  
    except:
        print("caught exception")

C:

//Here we register traceFunction to give us a hook into the python 
PyEval_SetProfile((Py_tracefunc)traceFunction, NULL);

//Abort interrupt reset to false
m_interruptRequest = false;

//Call the runLoop method
const char *szName = "runLoop";

PyObject_CallMethod(m_pyObject, szName, nullptr);

In the middle of this loop, I would like to inject an exception from C to abort the loop, but also handle the exception in the try/except in Python. As mentioned in the comments I am registering a profiling function in C called tracer be able to inject an exception from C. The following code below injects an exception and kills the program because it is not caught in the try/except.

//Tracer function callback hook into Python execution. Here we handle pause, resume and abort requests.

void PythonRunner::tracer(void)
{
    ...
    //Abort requested
    else if (true == interruptRequested())
    {
        PyErr_SetString(PyExc_Exception, "ABORT");


    }
}

Does anyone know how I can throw an exception from C to be handled in Python? Thanks

Update: With the help of @zwol and this article stackoverflow.com/questions/1420957/stopping-embedded-python I found a solution. The key was to add a pending call to python e.g

    int PythonRunner::tracer(PyObject *, _frame *, int, PyObject *)
    {
        ...
        //Abort requested
        else if (true == _instance->interruptRequested())
        {
            Py_AddPendingCall(&PythonRunner::raiseException, NULL);
        }
    return 0;
   }

int PythonRunner::raiseException(void *)
{
    PyErr_SetString(PyExc_KeyboardInterrupt, "Abort");
    return -1;
}

Answer 1

In general, to throw an exception using the CPython C API, you call one of the PyErr_Set* functions and then you return a special value from your API function. For functions that normally return a Python object, that special value is NULL; for other functions you have to check the documentation, which may only exist in the header files.

In this case, quoting pystate.h (from CPython 3.7):

/* Py_tracefunc return -1 when raising an exception, or 0 for success. */
typedef int (*Py_tracefunc)(PyObject *, struct _frame *, int, PyObject *);

That means you must declare PythonRunner::tracer with a matching signature, like so:

class PythonRunner {
    // ...
public:
    static int tracer(PyObject *, struct _frame *, int, PyObject *);
}

And then you can return −1 to signal an exception, or 0 to continue normally:

int PythonRunner::tracer(PyObject *, struct _frame *, int, PyObject *)
{
    ...
    // Abort requested?
    else if (interruptRequested())
    {
        PyErr_SetString(PyExc_Exception, "ABORT");
        return -1;
    }
    // Otherwise proceed normally.
    return 0;
}

Also, your call to PyEval_SetProfile will then no longer need a cast.

PyEval_SetProfile(PythonRunner::tracer, NULL);

I mention this because casting function pointers is almost always a bug in and of itself, and in this case, recognizing that could have clued you into what the problem was.

(For historical reasons, the Python "Extending and embedding" documentation's example code contains a bunch of unnecessary casts and functions with incorrect signatures. Whenever you copy code out of that document, remove all of the casts from it, and then carefully inspect the compiler's complaints, adjust type signatures, and put back only the casts that are unavoidable.)