cython - determining the number of items in pointer variable

Question

How would I determine the number of elements in a pointer variable in cython? I saw that in C one way seems to be sizeof(ptr)/sizeof(int), if the pointer points to int variables. But that doesn't seem to work in cython. E.g. when I tried to join two memory views into a single pointer like so:

from libc.stdlib cimport malloc, free
cdef int * join(int[:] a, int[:] b):

    cdef:
        int n_a = a.shape[0]
        int n_b = b.shape[0]
        int new_size =  n_a + n_b
        int *joined = <int *> malloc(new_size*sizeof(int))
        int i
    try:
        for i in range(n_a):
            joined[i] = a[i]

        for i in range(n_b):
            joined[n_a+i] = b[i]

        return joined
    finally:
        free(joined)

@cython.cdivision(True)
def join_memviews(int[:] n, int[:] m):

    cdef int[:] arr_fst = n
    cdef int[:] arr_snd = m
    cdef int *arr_new
    cdef int new_size

    arr_new = join(arr_fst,arr_snd)    
    new_size = sizeof(arr_new)/sizeof(int)

    return [arr_new[i] for i in range(new_size)]

I do not get the desired result when calling join_memviews from a python script, e.g.:

# in python
a = np.array([1,2])
b = np.array([3,4])
a_b = join_memviews(a,b)

I also tried using the types

DTYPE = np.int
ctypedef np.int_t DTYPE_t

as the arguement inside sizeof(), but that didn't work either.

Edit: The handling of the pointer variable was apparently a bit careless of me. I hope the following is fine (even though it might not be a prudent approach):

cdef int * join(int[:] a, int[:] b, int new_size):

    cdef:
        int *joined = <int *> malloc(new_size*sizeof(int))
        int i

    for i in range(n_a):
        joined[i] = a[i]

    for i in range(n_b):
        joined[n_a+i] = b[i]

    return joined

def join_memviews(int[:] n, int[:] m):

    cdef int[:] arr_fst = n
    cdef int[:] arr_snd = m
    cdef int *arr_new
    cdef int new_size = n.shape[0] + m.shape[0]

    try: 
        arr_new = join(arr_fst,arr_snd, new_size)    

        return [arr_new[i] for i in range(new_size)]
    finally:
        free(arr_new)

score 0 · Answer 1 · answered Jun 08 '17 at 16:40

0

You can't. It doesn't work in C either. sizeof(ptr) returns the amount of memory used to store the pointer (i.e. typically 4 or 8 depending on your system) rather than the length of the array. The lengths of your malloced arrays are something that you need to keep track of manually.

Additionally the following code is a recipe for disaster:

cdef int *joined = <int *> malloc(new_size*sizeof(int))
try:
   return joined
finally:
   free(joined)

The free happens immediately on function exit so that an invalid pointer is returned to the calling function.

You should be using properly managed Python arrays (either from numpy or the standard library array module) unless you absolutely can't avoid it.

answered Jun 08 '17 at 16:40

DavidW

29,336
6
55
86

thanks for these clarifications. Regarding the `try-finally` part, I thought that `joined` would be copied to the variable to which the function returns, before it is freed. But I haven't checked, so thanks for the warning. I edit the code that it should work properly, though it might not be a sensible approach. – user3820991 Jun 09 '17 at 10:02
Your edit looks like it should work. However, I'd be tempted to just use numpy to do the same task: `np.concatenate((a,b))` – DavidW Jun 09 '17 at 10:13
of course, if I would just want to join two arrays, I would use numpy. but I have the need to join memory views within a larger task implemented in cython. Because that happens quite a lot and because I saw [here](https://stackoverflow.com/questions/18462785/what-is-the-recommended-way-of-allocating-memory-for-a-typed-memory-view/21054369#21054369) that the malloc-approach offers fast initalization and access via iteration, I wanted to try it that way. Still, I should probably first try the save thing and than profile. – user3820991 Jun 09 '17 at 10:25

cython - determining the number of items in pointer variable

1 Answers1