4

So I am trying to write a C function that accepts a numpy array object, extracts the data, does some manipulations and returns another c array as a numpy array object. Everything works seamlessly and I use python wrappers which help easy manipulation on the python side. However, I am facing a memory leak. I have an output pointer of doubles that I malloc-ed and which I wrap into a Python array object just before returning it to the calling python function,

PyObject *arr;
int nd = 2;
npy_intp dims[] = {5, 10};
double *data = some_function_that_returns_a_double_star(x, y, z);

arr = PyArray_SimpleNewFromData(nd, dims, NPY_DOUBLE, (void *)data);
return arr;

However, this creates a memory leak, because data is never freed and I did some googling to find that this is a problem in such applications and solution is non-trivial. The most helpful resource that I found on this is given here. I could not implement the destructor that this page talks about from the given example. Can someone help me with this? More concretely I am looking for something like,

PyObject *arr;
int nd = 2;
npy_intp dims[] = {5, 10};
double *data = some_function_that_returns_a_double_star(x, y, z);

arr = PyArray_SimpleNewFromData(nd, dims, NPY_DOUBLE, (void *)data);
some_destructor_that_plug_memLeak_due_to_data_star(args);
return arr;
senior_mle
  • 809
  • 1
  • 10
  • 20

2 Answers2

7

The technique described in the link you didn't understand is a good one: create a Python object that knows how to free your memory when destroyed, and make it the base of the returned array.

It sounds like you might have been overwhelmed by the complexity of creating a new extension type. Fortunately, that's not necessary. Python comes with a type designed to perform arbitrary C-level cleanup when destroyed: capsules, which bundle together a pointer and a destructor function and call the destructor when the capsule is destroyed.

To create a capsule for your memory, first, we define a destructor function:

void capsule_cleanup(PyObject *capsule) {
    void *memory = PyCapsule_GetPointer(capsule, NULL);
    // I'm going to assume your memory needs to be freed with free().
    // If it needs different cleanup, perform whatever that cleanup is
    // instead of calling free().
    free(memory);
}

And you set a capsule as your array's base with

PyObject *capsule = PyCapsule_New(data, NULL, capsule_cleanup);
PyArray_SetBaseObject((PyArrayObject *) arr, capsule);
// Do not Py_DECREF the capsule; PyArray_SetBaseObject stole your
// reference.

And that should ensure your memory gets freed once it's no longer in use.

user2357112
  • 260,549
  • 28
  • 431
  • 505
  • Thanks a bunch ... that was really neat ... seems to have actually plugged my leak ... couldn't find this anywhere when I was searching for a fix. Very elegant. Just a couple of pointers though, (a) `void *memory = PyCapsule_GetPointer(capsule, NULL);` (b) when I compile the above code, I get the following warning - `passing argument 1 of ‘(int (*)(PyArrayObject *, PyObject *))*(PyArray_API + 2256u)’ from incompatible pointer type [-Wincompatible-pointer-types]` Any ideas on averting this? – senior_mle Oct 10 '18 at 03:45
  • @ArnabSanyal: Thanks for pointing out the missing argument; I'd forgotten capsule names were used for that. As for the pointer type warning, there was a missing cast. – user2357112 Oct 10 '18 at 04:20
  • You're welcome @user2357112. This was a real lifesaver though. Tested it. works like a charm !! – senior_mle Oct 10 '18 at 04:28
  • I found this in caffe library, maybe someone will help: "boost python expects a void (missing) return value, while import_array" "returns NULL for python3. import_array1() forces a void return value." – Roger Figueroa Quintero Aug 08 '19 at 09:46
7

While the PyCapsule approach works more generally, you can get numpy to free the memory in the array for you when it's garbage collected by setting the OWNDATA flag.

double *data = some_function_that_returns_a_double_star(x, y, z);
PyObject *arr = PyArray_SimpleNewFromData(nd, dims, NPY_DOUBLE, (void *)data);
PyArray_ENABLEFLAGS((PyArrayObject*) arr, NPY_ARRAY_OWNDATA);
Robin Betz
  • 69
  • 1
  • 2