Storing data using PyArray_NewFromDescr

Question

I use cython and I need to store the data as shown below. Earlier I used for loops to store the data from pus_image[0] into a 3D array but when running for n frames it created a bottleneck in performance. Hence I used PyArray_NewFromDescr to store which solves the bottleneck issue earlier faced. But the displayed images look different from the previous method, as I am not able to do increment _puc_image += aoiStride. Could anyone please help me solve this issue.

Code 1 :

    def LiveAquisition(self,nframes,np.ndarray[np.uint16_t,ndim = 3,mode = 'c']data):
    cdef:
        int available
        AT_64 sizeInBytes
        AT_64 aoiStride
        AT_WC string[20]
        AT_WC string1[20]
        AT_WC string2[20]
        AT_WC string3[20]
        unsigned char * pBuf
        unsigned char * _puc_image
        int BufSize
        unsigned int i, j, k, l = 0

    for i in range(nframes):
        pBuf = <unsigned char *>calloc(sizeInBytes, sizeof(unsigned char))
        AT_QueueBuffer(<AT_H>self.cameraHandle, pBuf, sizeInBytes)

        print "Frame number is :",
        print i
        response_code = AT_WaitBuffer(<AT_H>self.cameraHandle, &pBuf, &BufSize, 500)

        _puc_image = pBuf
        pus_image = <unsigned short*>pBuf
        for j in range(self.aoiWidth/self.hbin):
            pus_image = <unsigned short*>(_puc_image)
            for k in range(self.aoiHeight/self.vbin):
                data[l][j][k]  = pus_image[0]
                pus_image += 1
            _puc_image += aoiStride

    free(pBuf)
    return data

Code 2 : Using PyArray_NewFromDescr Prior to which its defined as :

from cpython.ref cimport PyTypeObject
from python_ref cimport Py_INCREF


cdef extern from "<numpy/arrayobject.h>":
object PyArray_NewFromDescr(PyTypeObject *subtype, np.dtype descr,int nd, np.npy_intp* dims,np.npy_intp*strides,void* data, int flags, object obj)

    def LiveAquisition(self,nframes,np.ndarray[np.uint16_t,ndim = 3,mode = 'c']data):
    cdef:
        int available
        AT_64 sizeInBytes
        AT_64 aoiStride
        AT_WC string[20]
        AT_WC string1[20]
        AT_WC string2[20]
        AT_WC string3[20]
        unsigned char * pBuf
        unsigned char * _puc_image
        int BufSize
        unsigned int i, j, k, l = 0
        np.npy_intp dims[2]
        np.dtype dtype = np.dtype('<B')


    for i in range(nframes):
        pBuf = <unsigned char *>calloc(sizeInBytes, sizeof(unsigned char))
        AT_QueueBuffer(<AT_H>self.cameraHandle, pBuf, sizeInBytes)
        print "Frame number is :",
        print i
        response_code = AT_WaitBuffer(<AT_H>self.cameraHandle, &pBuf, &BufSize, 500)

        Py_INCREF(dtype)
        dims[0] = self.aoiWidth
        dims[1] = self.aoiHeight
        data[i,:,:] = PyArray_NewFromDescr(<PyTypeObject *> np.ndarray, np.dtype('<B'), 2,dims, NULL,pBuf, np.NPY_C_CONTIGUOUS, None)

    free(pBuf)
    return data

Probably minor point, but for code 1, I notice you use `data[l][j][k]` instead of `data[l, j, k]`. Not 100% sure if it matters, but does changing that change performance in any way? — CodeSurgeon, May 28 '18 at 12:45
You're freeing the last frame (and no others) when numpy owns the data. I also suspect you're not handling "steals a reference to descr" correctly. Why not just get a pointer from an array allocated by numpy rather than allocating it yourself? — DavidW, May 28 '18 at 21:57
@CodeSugeon, yes changing to `data[l, j, k]` does make a difference.Thank you But I still want to figure out how to make method 2 work — aishwarya selvaraj, May 29 '18 at 07:50
@DavidW, could you please elaborate a bit. A pseudo-code would be great! — aishwarya selvaraj, May 29 '18 at 07:52
`pBuff = &data[i,0,0]` should allow your capture functions to write directly into data (I think...) — DavidW, May 29 '18 at 08:05

score 1 · Answer 1 · answered Jun 04 '18 at 16:57

There's a few large errors in the way you're doing this. However, what you're doing is totally unnecessary, and there's a much simpler approach. You can simply allocate the data using Numpy, and get the address of the first element of that array:

# earlier
cdef unsigned char[:,::1] p
# in loop
p = np.array((self.aoiWidth,self.aoiHeight),dtype=np.uint8)
pbuf = &p[0,0] # address of first element of p

# code goes here
data[i,:,:] = p

Errors in what you're doing:

pBuf = <unsigned char *>calloc(sizeInBytes, sizeof(unsigned char))

Here, sizeInBytes is uninitialized, and therefore the size you allocate with be arbitrary.
PyArray_NewFromDescr steals a reference to the descr argument. This means that it does not increment the reference count of the argument. The line
```
PyArray_NewFromDescr(<PyTypeObject *> np.ndarray, np.dtype('<B'), ...)
```
will be translated as Cython to something like
```
temp_dtype = np.dtype('<B') # refcount 1
PyArray_NewFromDescr(<PyTypeObject *> np.ndarray, temp_dtype, ...)
# temp_dtype refcount is still 1
Py_DECREF(temp_dtype) # Cython's own cleanup
# temp_dtype has now been destroyed, but is still being used by your array
```
It looks like you copied some code that dealt with this correctly (Py_INCREF(dtype), which was then passed to PyArray_NewFromDescr), but chose to ignore that and create your own temporary object.
PyArray_NewFromDescr does not own the data. Therefore you are responsible for deallocating it once it has been used (and only when you're sure it's no longer needed). You only do one free, after the loop, so you are leaking almost all the memory you allocated. Either put the free in the loop, or modify the OWNDATA flag to give your new array ownership of your array.

In summary, unless you have a good understanding of the Python C API I recommend don't using PyArray_NewFromDescr and using numpy arrays to allocate your data instead.

Storing data using PyArray_NewFromDescr

1 Answers1