Inspecting memory leaks in C++ with embedded Python

Question

I am writing a C++ code that embeds a Python module. I'm importing the module just once at the beginning; then, the C++ code repeatedly calls a single function from it. To do so, I've used the following syntax (here reporting a minimum working example, assuming the Python function takes two float arguments and returns a float value):

int main() {
   
    // Initialise the Python interpreter
    Py_Initialize();

    PyObject *pFunc = NULL, *pModule = NULL;

    const char* module_name   = "mod_name";
    const char* function_name = "fun_name";

    // Importing module
    pModule = PyImport_ImportModule(module_name);

    // Sanity check on module pointer
    if (pModule == NULL) {
        PyErr_Print();
        std::cout << "ERROR importing module" << std::endl;
        std::exit(1);
    }

    // Import function from Python module 
    pFunc = PyObject_GetAttrString(pModule, function_name);
    if (pFunc == NULL) {
        Py_CLEAR(pModule);
        PyErr_Print();
        std::cout << "ERROR getting test function" << std::endl;
        std::exit(1);
    }

    // Free module pointer
    Py_CLEAR(pModule);


    double val1 = 1.5;
    double val2 = 2.5;

    for (int i = 0; i < 10; i++){
        
        // Specify inputs of the Python function that has to be called
        pArgs = PyTuple_Pack(2, PyFloat_FromDouble(val1), PyFloat_FromDouble(val2) );
 
         
        // Checking that the tuple packing was performed correctly
        if (pArgs == NULL) {
            PyErr_Print();
            std::cout << "ERROR packing arguments" << std::endl;
            std::exit(1);
        } 
        
   
        // Call the Python function and store the returned value in pValue
        pValue = PyObject_CallObject(pFunc, pArgs);

        // Checking that the function call was performed correctly
        if (pValue == NULL) {
            Py_CLEAR(pArgs);
            PyErr_Print();
            std::cout << "ERROR getting value from function" << std::endl;
            std::exit(1);
         }
    
        //SUCCESSFULLY COMPLETED PYTHON CALL
        double res= PyFloat_AsDouble(pValue);
    
        // Free all the allocated memory
        Py_CLEAR(pArgs);
        Py_CLEAR(pValue);
   
    } 

    // Free the shared pointer to python function
    Py_CLEAR(pFunc);
    Py_Finalize();
    return 0;
}

I have profiled my C++ executable using valgrind, to inspect for memory leaks. While when I isolate the C++ sections of the code everything works fine, as soon as I run the complete code with the embedded Python section, I get memory leaks. Please see below for an example loss report from the original code: valgrind loss report

I know that valgrind may give odd results when interacting with Python code, so I am reaching out here to see if there are any best practices to inspect for memory leaks when embedding Python in C++.

Also, this is my first question here, so if any useful info is missing please do not hesitate in asking for it! Thanks in advance

Answer 1

The issue that you are having is that valgrind understands the allocations done via malloc but it doesn't understand allocations from the python arena. So if a malloc'd allocation is referenced only from a python allocation valgrind won't pick up on that and will consider the malloc'd allocation to be leaked.

My recommendation is that you use https://github.com/vmware/chap because it can recognize both kinds of allocations.

Run your program without valgrind, but at some point before your program finishes gather a live core, either using gcore or by running "generate" from gdb at a break point.

Then start chap and try the following from the chap prompt:

count leaked

If that shows a non-zero value, check out https://github.com/vmware/chap/blob/master/USERGUIDE.md for further advice on how to analyze the leak.