Efficient math ops on small arrays in python with cython

Question

I use numpexpr for fast math on large arrays but if the size of the array is less than the CPU cache, writing my code in Cython using simple array math is way faster, especially, if the function is called multiple times.

The issue is, how do you work with arrays in Cython, or more explicitly: is there a direct interface to Python's array.array type in Cython? What I would like to do is something like this (simple example)

cpdef array[double] running_sum(array[double] arr):
    cdef int i 
    cdef int n = len(arr)
    cdef array[double] out = new_array_zeros(1.0, n)
    ... # some error checks
    out[0] = arr[0]
    for i in xrange(1,n-1):
        out[i] = out[i-1] + arr[i]

    return(out)

I first tried using Cython numpy wrapper and worked with the ndarrays but it seems that creating them is very costly for small 1D arrays, compared with creating a C array with malloc (but memory handling becomes a pain).

Thanks!

You should really split this into two separate questions since each part is distinct. This will ensure that the answers are clearly associated with a particular question and will increase readability for users who reference this question and its answers in the future. — JoshAdel, Mar 19 '11 at 03:05
Thanks. Split the question into two parts - second part: http://stackoverflow.com/questions/5359880/hdf5-and-ndarray-append-time-efficient-approach-for-large-data-sets — chronos, Mar 19 '11 at 03:38
yes, numpy new-array is slower than malloc; but do you really have to create / delete a lot, can you not create numpy arrays once at the beginning and reuse them ? Also, timeits on the above vs. np.cumsum might be useful (what's "small" -- 10, 100 ?) — denis, Mar 21 '11 at 12:06
For posterities' sake, I answered a very similar question [here](http://stackoverflow.com/questions/13959281/speeding-up-python-code-with-cython/13962753#13962753). Recent cython (0.17+) has a lot of [good features](http://docs.cython.org/src/userguide/memoryviews.html) for working with arrays and numpy.ndarrays and everything else that supports the buffer interface. — John Tyree, Jan 19 '13 at 23:51

score 5 · Answer 1 · answered Mar 30 '11 at 17:37

You can roll your simple own with basic functions and checks here is a mockup to start:

from libc.stdlib cimport malloc,free

cpdef class SimpleArray:
    cdef double * handle
    cdef public int length
    def __init__(SimpleArray self, int n):
        self.handle = <double*>malloc(n * sizeof(double))
        self.length = n
    def __getitem__(self, int idx):
        if idx < self.length:
            return self.handle[idx]
        raise ValueError("Invalid Idx")
    def __dealloc__(SimpleArray self):
        free(self.handle) 

cpdef SimpleArray running_sum(SimpleArray arr):
    cdef int i 
    cdef SimpleArray out = SimpleArray(arr.length)

    out.handle[0] = arr.handle[0]
    for i from 1 < i < arr.length-1:
        out.handle[i] = out.handle[i-1] + arr.handle[i]
    return out

can be used as

>>> import test
>>> simple = test.SimpleArray(100)
>>> del simple
>>> test.running_sum(test.SimpleArray(100))
<test.SimpleArray object at 0x1002a90b0>

Efficient math ops on small arrays in python with cython

1 Answers1

Linked