Accessing a Python traceback from the C API

Question

I'm having some trouble figuring out the proper way to walk a Python traceback using the C API. I'm writing an application that embeds the Python interpreter. I want to be able to execute arbitrary Python code, and if it raises an exception, to translate it to my own application-specific C++ exception. For now, it is sufficient to extract just the file name and line number where the Python exception was raised. This is what I have so far:

PyObject* pyresult = PyObject_CallObject(someCallablePythonObject, someArgs);
if (!pyresult)
{
    PyObject* excType, *excValue, *excTraceback;
    PyErr_Fetch(&excType, &excValue, &excTraceback);
    PyErr_NormalizeException(&excType, &excValue, &excTraceback);

    PyTracebackObject* traceback = (PyTracebackObject*)traceback;
    // Advance to the last frame (python puts the most-recent call at the end)
    while (traceback->tb_next != NULL)
        traceback = traceback->tb_next;

    // At this point I have access to the line number via traceback->tb_lineno,
    // but where do I get the file name from?

    // ...       
}

Digging around in the Python source code, I see they access both the filename and module name of the current frame via the _frame structure, which looks like it is a privately-defined struct. My next idea was to programmatically load the Python 'traceback' module and call its functions with the C API. Is this sane? Is there a better way to access a Python traceback from C?

[PyErr_Fetch](https://docs.python.org/2/c-api/exceptions.html#c.PyErr_Fetch) produces memory leak (depending on the implementation it may be important) — alex, Feb 28 '16 at 14:55
What's the purpose of "PyTracebackObject* traceback = (PyTracebackObject*)traceback;"? I think you mean "PyTracebackObject* traceback = (PyTracebackObject*)excTraceback;". — aquirdturtle, May 29 '16 at 03:00

score 18 · Answer 1 · edited Oct 30 '16 at 19:54

This is an old question but for future reference, you can get the current stack frame from the thread state object and then just walk the frames backward. A traceback object isn't necessary unless you want to preserve the state for the future.

For example:

PyThreadState *tstate = PyThreadState_GET();
if (NULL != tstate && NULL != tstate->frame) {
    PyFrameObject *frame = tstate->frame;

    printf("Python stack trace:\n");
    while (NULL != frame) {
        // int line = frame->f_lineno;
        /*
         frame->f_lineno will not always return the correct line number
         you need to call PyCode_Addr2Line().
        */
        int line = PyCode_Addr2Line(frame->f_code, frame->f_lasti);
        const char *filename = PyString_AsString(frame->f_code->co_filename);
        const char *funcname = PyString_AsString(frame->f_code->co_name);
        printf("    %s(%d): %s\n", filename, line, funcname);
        frame = frame->f_back;
    }
}

sadly this no longer compiles with python3.11 – Goblinhack Dec 02 '22 at 10:09 — Goblinhack, Dec 02 '22 at 10:09

score 14 · Answer 2 · edited Oct 29 '21 at 17:36

14

I prefer calling into python from C:

err = PyErr_Occurred();
if (err != NULL) {
    PyObject *ptype, *pvalue, *ptraceback;
    PyObject *pystr, *module_name, *pyth_module, *pyth_func;
    char *str;

    PyErr_Fetch(&ptype, &pvalue, &ptraceback);
    pystr = PyObject_Str(pvalue);
    str = PyString_AsString(pystr);
    error_description = strdup(str);

    /* See if we can get a full traceback */
    module_name = PyString_FromString("traceback");
    pyth_module = PyImport_Import(module_name);
    Py_DECREF(module_name);

    if (pyth_module == NULL) {
        full_backtrace = NULL;
        return;
    }

    pyth_func = PyObject_GetAttrString(pyth_module, "format_exception");
    if (pyth_func && PyCallable_Check(pyth_func)) {
        PyObject *pyth_val;

        pyth_val = PyObject_CallFunctionObjArgs(pyth_func, ptype, pvalue, ptraceback, NULL);

        pystr = PyObject_Str(pyth_val);
        str = PyString_AsString(pystr);
        full_backtrace = strdup(str);
        Py_DECREF(pyth_val);
    }
}

edited Oct 29 '21 at 17:36

Will Da Silva

6,386
2
27
52

answered Apr 09 '13 at 16:30

Ian Main

149
1
2

You're missing some Py_DECREFs here ... `pystr` needs to be decref'd after each call to `PyObject_Str`, and `pyth_module` also needs to be decref'd. – Charles Salvia Jul 27 '14 at 19:34
To clarify, this code is an implementation of myTraceBackMunger() proposed by Ned Batchholder's answer. The point is, that even though an exception might have occurred, the Python interpreter can still be used, and should be used because it's handling of the traceback is at a higher level, you don't need to understand the details. However, if you do want to use C, Jason McCampbell's answer seems like the simplest one, not even using a traceback object, just the underlying stack of frames. – bootchk Sep 13 '14 at 12:16
With this code I got an error when calling `PyObject_CallFunctionObjArgs` - `pvalue` had wrong type. So I have added `PyErr_NormalizeException` after `PyErr_Fetch` as Bartosz Kosarzycki did in his answer and it works now. – Michał Walenciak Mar 03 '18 at 09:31

score 9 · Accepted Answer · answered Nov 25 '09 at 12:21

I've discovered that _frame is actually defined in the frameobject.h header included with Python. Armed with this plus looking at traceback.c in the Python C implementation, we have:

#include <Python.h>
#include <frameobject.h>

PyTracebackObject* traceback = get_the_traceback();

int line = traceback->tb_lineno;
const char* filename = PyString_AsString(traceback->tb_frame->f_code->co_filename);

But this still seems really dirty to me.

Ned Batchelder · Answer 4 · 2009-11-25T12:20:21.243

6

One principal I've found useful in writing C extensions is to use each language where it's best suited. So if you have a task to do that would be best implemented in Python, implement in Python, and if it would be best implemented in C, do it in C. Interpreting tracebacks is best done in Python for two reasons: first, because Python has the tools to do it, and second, because it isn't speed-critical.

I would write a Python function to extract the info you need from the traceback, then call it from C.

You could even go so far as to write a Python wrapper for your callable execution. Instead of invoking someCallablePythonObject, pass it as an argument to your Python function:

def invokeSomeCallablePythonObject(obj, args):
    try:
        result = obj(*args)
        ok = True
    except:
        # Do some mumbo-jumbo with the traceback, etc.
        result = myTraceBackMunger(...)
        ok = False
    return ok, result

Then in your C code, call this Python function to do the work. The key here is to decide pragmatically which side of the C-Python split to put your code.

edited Nov 25 '09 at 12:20

answered Nov 25 '09 at 12:13

Ned Batchelder

364,293
75
561
662

Not sure I understand how this helps. I am not writing an extension module, but rather embedding the interpreter. So to implement your solution (if I'm understand you right) I would have to write a blob of Python code and store it in my C++ code as a string. Then at some point I would have to compile the code, create a function out of it, then call the function via PyObject_CallObject. This seems like a ton of work compared to just examining the native stack frame structures in C. – cwick Nov 26 '09 at 08:10
This advice to split between the languages pragmatically based on their strengths makes a lot of sense, but I feel wary about trying to execute arbitrary Python to process the error state from executing some other arbitrary Python. – caps Apr 27 '17 at 18:56
Note that Python now has a `traceback` library suited for this approach. https://docs.python.org/3/library/traceback.html – soegaard Jul 08 '22 at 18:56

score 4 · Answer 5 · answered Apr 12 '17 at 14:46

I used the following code to extract Python exception's error body. strExcType stores the exception type and strExcValue stores the exception body. Sample values are:

strExcType:"<class 'ImportError'>"
strExcValue:"ImportError("No module named 'nonexistingmodule'",)"

Cpp code:

if(PyErr_Occurred() != NULL) {
    PyObject *pyExcType;
    PyObject *pyExcValue;
    PyObject *pyExcTraceback;
    PyErr_Fetch(&pyExcType, &pyExcValue, &pyExcTraceback);
    PyErr_NormalizeException(&pyExcType, &pyExcValue, &pyExcTraceback);

    PyObject* str_exc_type = PyObject_Repr(pyExcType);
    PyObject* pyStr = PyUnicode_AsEncodedString(str_exc_type, "utf-8", "Error ~");
    const char *strExcType =  PyBytes_AS_STRING(pyStr);

    PyObject* str_exc_value = PyObject_Repr(pyExcValue);
    PyObject* pyExcValueStr = PyUnicode_AsEncodedString(str_exc_value, "utf-8", "Error ~");
    const char *strExcValue =  PyBytes_AS_STRING(pyExcValueStr);

    // When using PyErr_Restore() there is no need to use Py_XDECREF for these 3 pointers
    //PyErr_Restore(pyExcType, pyExcValue, pyExcTraceback);

    Py_XDECREF(pyExcType);
    Py_XDECREF(pyExcValue);
    Py_XDECREF(pyExcTraceback);

    Py_XDECREF(str_exc_type);
    Py_XDECREF(pyStr);

    Py_XDECREF(str_exc_value);
    Py_XDECREF(pyExcValueStr);
}

moof2k · Answer 6 · 2014-12-18T16:13:29.880

I had reason to do this recently while writing an allocation tracker for numpy. The previous answers are close but frame->f_lineno will not always return the correct line number--you need to call PyFrame_GetLineNumber(). Here's an updated code snippet:

#include "frameobject.h"
...

PyFrameObject* frame = PyEval_GetFrame();
int lineno = PyFrame_GetLineNumber(frame);
PyObject *filename = frame->f_code->co_filename;

The full thread state is also available in the PyFrameObject; if you want to walk the stack keep iterating on f_back until it's NULL. Checkout the full data structure in frameobject.h: http://svn.python.org/projects/python/trunk/Include/frameobject.h

See also: https://docs.python.org/2/c-api/reflection.html

score 0 · Answer 7 · edited Dec 02 '21 at 10:26

You can access Python traceback similar to tb_printinternal function. It iterates over PyTracebackObject list. I have tried also suggestions above to iterate over frames, but it does not work for me (I see only the last stack frame).

Excerpts from CPython code:

static int
tb_displayline(PyObject *f, PyObject *filename, int lineno, PyObject *name)
{
    int err;
    PyObject *line;

    if (filename == NULL || name == NULL)
        return -1;
    line = PyUnicode_FromFormat("  File \"%U\", line %d, in %U\n",
                                filename, lineno, name);
    if (line == NULL)
        return -1;
    err = PyFile_WriteObject(line, f, Py_PRINT_RAW);
    Py_DECREF(line);
    if (err != 0)
        return err;
    /* ignore errors since we are not able to report them, are we? */
    if (_Py_DisplaySourceLine(f, filename, lineno, 4))
        PyErr_Clear();
    return err;
}

static int
tb_printinternal(PyTracebackObject *tb, PyObject *f, long limit)
{
    int err = 0;
    long depth = 0;
    PyTracebackObject *tb1 = tb;
    while (tb1 != NULL) {
        depth++;
        tb1 = tb1->tb_next;
    }
    while (tb != NULL && err == 0) {
        if (depth <= limit) {
            err = tb_displayline(f,
                                 tb->tb_frame->f_code->co_filename,
                                 tb->tb_lineno,
                                 tb->tb_frame->f_code->co_name);
        }
        depth--;
        tb = tb->tb_next;
        if (err == 0)
            err = PyErr_CheckSignals();
    }
    return err;
}

score 0 · Answer 8 · answered Dec 02 '22 at 10:36

As of python 3.11, accessing the frame objects seems to need a different approach. Anyway, this works in 3.11, hth someone

py_err(void)
{
  PyObject *err = PyErr_Occurred();
  if (! err) {
    return;
  }

  PyObject *ptype, *pvalue, *pbacktrace, *pyobj_str;
  PyObject *ret, *list, *string;
  PyObject *mod;
  char     *py_str;

  PyErr_Fetch(&ptype, &pvalue, &pbacktrace);
  PyErr_NormalizeException(&ptype, &pvalue, &pbacktrace);
  PyErr_Display(ptype, pvalue, pbacktrace);
  PyTraceBack_Print(pbacktrace, pvalue);

  pyobj_str = PyObject_Str(pvalue);
  py_str    = py_obj_to_string(pyobj_str);
  printf("%s", py_str);
  myfree(py_str);

  mod  = PyImport_ImportModule("traceback");
  list = PyObject_CallMethod(mod, "format_exception", "OOO", ptype, pvalue, pbacktrace);
  if (list) {
    string = PyUnicode_FromString("\n");
    ret    = PyUnicode_Join(string, list);
    Py_DECREF(list);
    Py_DECREF(string);

    py_str = py_obj_to_string(ret);
    printf("%s", py_str);
    myfree(py_str);

    Py_DECREF(ret);
  }

  PyErr_Clear();
}

and you will probably need this too

char *py_obj_to_string(const PyObject *py_str)
{ 
  PyObject *py_encstr;
  char     *outstr = nullptr;
  char     *str;

  py_encstr = nullptr;
  str       = nullptr;

  if (! PyUnicode_Check((PyObject *) py_str)) {
    goto err_out;
  }
  
  py_encstr = PyUnicode_AsEncodedString((PyObject *) py_str, "utf-8", nullptr);
  if (! py_encstr) {
    goto err_out;
  }

  str = PyBytes_AS_STRING(py_encstr);
  if (! str) {
    goto err_out;
  }

  outstr = strdup(str);

err_out:
  
  if (py_encstr) {
    Py_XDECREF(py_encstr);
  }
  
  return outstr;
}

actual working code if someone needs it can be found in my larger project https://github.com/goblinhack/zorbash

Accessing a Python traceback from the C API

8 Answers8

Linked