Cython's `cdef` raises a NameError where a `def` works just fine

Question

Maybe I am understanding the cdef for function definition incorrectly. E.g., assume I want to write a function to convert a Python list to a C array:

%%cython
cimport cython
from libc.stdlib cimport malloc, free
cdef int* list_to_array(a_list):
    """ Converts a Python list into a C array """
    cdef int *c_array
    cdef int count, n
    c_array = <int *>malloc(len(a_list)*cython.sizeof(int))
    count = len(a_list)
    for i in range(count):
        c_array[i] = a_list[i]
    return c_array

when I call the function now via

list_to_array([1,2,3])

I get a

---------------------------------------------------------------------------
NameError                                 Traceback (most recent call last)
<ipython-input-32-8f3f777d7883> in <module>()
----> 1 list_to_array([1,2,3])

NameError: name 'list_to_array' is not defined

However, when I just use the def, the function can be called (although it doesn't return what I want, it is just for illustrating my problem...)

%%cython
cimport cython
from libc.stdlib cimport malloc, free
def list_to_array1(a_list):
    """ Converts a Python list into a C array """
    cdef int *c_array
    cdef int count, n
    c_array = <int *>malloc(len(a_list)*cython.sizeof(int))
    count = len(a_list)
    for i in range(count):
        c_array[i] = a_list[i]
    return 1


list_to_array1([1,2,3])

1

When I tried to use cpdef instead of cdef, I encounter a different issue:

Error compiling Cython file:
------------------------------------------------------------
...
cimport cython
from libc.stdlib cimport malloc, free
cpdef int* list_to_carray(a_list):
     ^
------------------------------------------------------------

/Users/sebastian/.ipython/cython/_cython_magic_c979dc7a52cdfb492e901a4b337ed2d2.pyx:3:6: Cannot convert 'int *' to Python object

score 1 · Answer 1 · edited May 23 '17 at 12:05

1

Citing the docs, "The cdef statement is used to make C level declarations"

Then if you scroll a bit down here, you see that cdef functions can not be called from python, hence your NameError. Try using cpdef.

Note that if you plan to use that function in python code it will leak memory. You may also want to have a look at this answer on how/why you should return pass a list to/from cython (disclaimer: the answer is mine) to avoid the leakage.

EDIT, in reply to updated question:

The error once you introduce cpdef happens because a pointer cannot be converted to a python object in a trivial way. Cython does the hard work for you in the simplest cases, see here. The question you should ask here is why you want to return a C pointer to the python environment, that does not provide pointers.

edited May 23 '17 at 12:05

Community

1
1

answered May 14 '14 at 15:30

gg349

21,996
5
54
64

Thanks, I have a second function `array_to_pythonlist` that would take care of the memory leak via `libc.stdlib.free(c_array)` but I didn't mention it because it was not directly related to the issue of the question – May 14 '14 at 16:38
Abput cpdef: I tried it before, and got a `Cannot convert 'int *' to Python object` , I will post the complete error in the question above. Thanks a lot for your help! – May 14 '14 at 16:40
Ok. you're however incurring in 2 copy overhead. If you pass in to cython a numpy array you don't have to copy to/from a c allocated block. – gg349 May 14 '14 at 16:41
The point why I am doing this is to compare it to a numpy array approach (in terms of a benchmark). When I am using numpy arrays as input, I noticed that this approach is "slower" than using C arrays, I have the benchmark here if you are interested: http://htmlpreview.github.io/?https://github.com/rasbt/One-Python-benchmark-per-day/blob/master/htmls/day4_python_cython_numba.html – May 14 '14 at 16:45
I still don't see why you need to return a C pointer to python. Open a new question on the performance of the 2 approaches, the question is interesting IMHO. I would try using `xrange` instead of `range`, and would try using `cpdef` instead of `def` where you declare the 2 tests `cython_bubblesort_clist` and `cython_bubblesort_numpy`. I think you should also try [this](http://docs.cython.org/src/tutorial/numpy.html#efficient-indexing) for efficient indexing. – gg349 May 14 '14 at 17:00
and disable bound checking, writing `@cython.boundscheck(False)` just before the definition of the 2 functions – gg349 May 14 '14 at 17:02
about xrange: I am using Python 3.x - in Python 3.x range() has been implemented as xrange() – May 14 '14 at 17:36
about why I want to return a c-pointer: I want to pass the c-array to the bubblesort cython implementation to benchmark it without doing the conversion from python list to c array internally in the function itself – May 14 '14 at 17:38
@cython.boundscheck(False) didn't help, `cpdef` still doesn't work, and I also still get the NameError with `cdef` (but as you said `cdef` wouldn't work either way within Python) – May 14 '14 at 17:40
in reply to your first comment of the 2. The cython implementation is not a python function, it should be declared with `cpdef` and can then accept a pointer. Your 2nd comment is inconclusive. You insisted not asking a new question about the performance of your benchmark, you have my -1. – gg349 May 14 '14 at 17:44
sorry, I just posted the question about the benchmark here: http://stackoverflow.com/questions/23661636/poorer-performance-of-cython-with-numpy-array-memoryview-compared-to-c-arrays – May 14 '14 at 17:48
`cpdef` causes the code to not compile: `@cython.boundscheck(False) cpdef int* list_to_carray(a_list):` with an error `Cannot convert 'int *' to Python object` – May 14 '14 at 17:50
the cpdef should be used on the cython implementation of the sort, not on `list_to_carray`. – gg349 May 14 '14 at 17:58

Cython's `cdef` raises a NameError where a `def` works just fine

1 Answers1