How to register a destructor for a C-allocated numpy array?

Question

I want to allocate numbers for a numpy array in C/C++, and pass them to python as a numpy array. That I can do with PyArray_SimpleNewFromData.

The problem is that I also want to register a function that should be invoked from Python when the numpy array reference counter reaches zero, and that would invoke some destructor semantics on the C side... Here is a pseudo-example of what I need:

 float* arr; PyObject* np_arr; void (*destructor)(float* arr);
 // ... C-allocate array on arr, ...
 // ...
 // ... initialize destructor with some suitable value, and then:
 np_arr = /* ... create the array to wrap arr, 
             and to use destructor on some meaningful way ... */

is there a simple way of doing so?

Answer 1

The idea is to create a Python object that knows how to free your memory when destroyed and make it the base of the returned C-allocated numpy array. This sounds tricky but it can be easily achieved via something known as capsules in python. Let me give an example,

Suppose that you have the following code,

PyObject *arr;
int nd = 2;
npy_intp dims[] = {5, 10};
double *data = some_function_that_returns_a_double_star(x, y, z);

arr = PyArray_SimpleNewFromData(nd, dims, NPY_DOUBLE, (void *)data);
return arr;

There is an obvious memory leak here since you cannot free data until arr is deleted as it says here in the red warning box. Fixing this, on the other hand, is easy. Define a function which is basically a destructor function that knows how to do garbage collection.

void capsule_cleanup(PyObject *capsule) {
    void *memory = PyCapsule_GetPointer(capsule, NULL);
    // Use your specific gc implementation in place of free if you have to
    free(memory);
}

Now augment your code as,

PyObject *arr;
int nd = 2;
npy_intp dims[] = {5, 10};
double *data = some_function_that_returns_a_double_star(x, y, z);

arr = PyArray_SimpleNewFromData(nd, dims, NPY_DOUBLE, (void *)data);
PyObject *capsule = PyCapsule_New(data, NULL, capsule_cleanup);
// NULL can be a string but use the same string while calling PyCapsule_GetPointer inside capsule_cleanup
PyArray_SetBaseObject((PyArrayObject *) arr, capsule);
return arr;

There is no need to Py_DECREF the capsule. The function PyArray_SetBaseObject steals reference.

Hope this helps!