Crash of Jupyter due to the use of PyArray_ENABLEFLAGS

Question

I try to convert a C double* in numpy array on Cython but I didn't succeed yet. I found these usefull links: Force NumPy ndarray to take ownership of its memory in Cython https://github.com/numpy/numpy/issues/8253

But everytime I use the following .pyx file with Cython it makes Jupyter crash (dead kernel):

.c file:

#include<stdlib.h>
#include "test.h"

double* test1(int n) {
  double* A=(double*)calloc(n,sizeof(double));
  int i;
  for (i=0;i<n;i++) {
    A[i]=i+0.5; }
  return(A); }

I also try with malloc with the same result.

.pyx file:

cimport c_test
import numpy as np
cimport numpy as np

np.import_array()

ctypedef np.float64_t DTYPE_t

cdef extern from "numpy/arrayobject.h":
    void PyArray_ENABLEFLAGS(np.ndarray arr, int flags)

cdef data_to_numpy_array_with_spec(void * ptr, np.npy_intp N, int t):
    cdef np.ndarray[DTYPE_t, ndim=1] arr = np.PyArray_SimpleNewFromData(1, &N, t, ptr)
    PyArray_ENABLEFLAGS(arr, np.NPY_OWNDATA)
    return arr

def test(n):
    cdef double* t1
    t=data_to_numpy_array_with_spec(c_test.test1(n),n,np.NPY_FLOAT64)
    return(t)

The c_fct.test1 function return a C double* of n double that I want to convert into a numpy array which own the data to avoid memory leak. Without the line PyArray_ENABLEFLAGS(t, np.NPY_ARRAY_OWNDATA) everything works fine but the memory is not deallocated when the numpy array is destroyed.

jupyter notebook:

import cy_test as ct
ct.test(1000)

c_test.pxd file:

cdef extern from "test.h":
    double* test1(int n)

My setup.py file looks like this:

from setuptools import setup
from setuptools.extension import Extension
from Cython.Distutils import build_ext
from Cython.Build import cythonize
import numpy as np

ext_modules = cythonize([Extension("cy_test", ["cy_test.pyx","test.c"])])


setup(
  name = 'Hello world app',
  cmdclass = {'build_ext': build_ext},
  ext_modules = ext_modules,
  include_dirs=[np.get_include()]
)

I think that my problem may come from the same reason as in the github link but I don't see how to solve it (in my case the C pointer already exist so I think I can't use the same solution).

I work on Windows 10 with Anaconda 64bits, here are the detail of the version: 3.7.1 (default, Dec 10 2018, 22:54:23) [MSC v.1915 64 bit (AMD64)]

I'm more interested in the test.c file it uses. It's possible that it's a more complicated problem, but it's worth being sure there's nothing simple in the code that allocates the memory. — DavidW, Jan 19 '19 at 23:17
The other information that might be useful is information about your system - Windows/Linux? What compiler? The GitHub link implies that this might be a compiler mismatch on Windows - if so I don't think I know how to fix it, but it'll need that information for anyone to help. — DavidW, Jan 19 '19 at 23:25
Right, I work on Windows 10, with Anaconda 64 bits (details: 3.7.1 (default, Dec 10 2018, 22:54:23) [MSC v.1915 64 bit (AMD64)] ) — pdzx, Jan 19 '19 at 23:45

user2357112 · Accepted Answer · 2019-01-20T00:19:55.130

2

As explained in the Github issue that you linked, NPY_OWNDATA is only safe to use with memory allocated through the same allocator NumPy itself uses. This allocator can be accessed through the PyDataMem_* functions. If your memory does not come from this allocator, you cannot use NPY_OWNDATA.

Don't try to force an array to take ownership of arbitrary memory you give it. Instead, set the array's base to an object that knows how to perform the correct cleanup, using PyArray_SetBaseObject. A capsule may be a convenient object to use.

edited Jan 20 '19 at 00:19

answered Jan 20 '19 at 00:07

user2357112

260,549
28
431
505

1

One good option for memory ownership might be [a Cython array, which can have a callback set for destruction](https://cython.readthedocs.io/en/latest/src/userguide/memoryviews.html#cython-arrays). These are readily converted to a numpy array by `np.asarray(cython_array_variable)`.(since they have the Python buffer interface) – DavidW Jan 20 '19 at 15:11

score 0 · Answer 2 · answered Jan 20 '19 at 15:51

Ok thanks to you it's done, all I had to do was to modified the .pyx file like this:

cimport c_test
import numpy as np
cimport numpy as np
from libc.stdlib cimport free

np.import_array()

ctypedef void (*PyCapsule_Destructor)(void*)

cdef extern from "numpy/arrayobject.h":
    void* PyCapsule_GetPointer(void* capsule, const char *name)
    void* PyCapsule_New(void *pointer, const char *name, PyCapsule_Destructor destructor)
    int PyArray_SetBaseObject(np.ndarray arr, void* obj)


cdef void capsule_cleanup(void* capsule):
    cdef void *memory = PyCapsule_GetPointer(capsule, NULL)
    free(memory)

def test(n):
  cdef np.ndarray arr
  cdef int nd = 1
  cdef np.npy_intp shape[1]
  shape[0] = <np.npy_intp> n
  cdef double *data = c_test.test1(n)
  arr = np.PyArray_SimpleNewFromData(nd, shape, np.NPY_DOUBLE, data)
  cdef void* capsule = PyCapsule_New(data, NULL, capsule_cleanup)
  PyArray_SetBaseObject( arr, capsule)
  return(arr)

Crash of Jupyter due to the use of PyArray_ENABLEFLAGS

2 Answers2