7

I am writing a Python wrapper for a function call with the signature

double** foo(double** arrayIn, int dim1, int dim2);

and need to build arrayIn inside my Python wrapper. One possible solution for this is given here. However, since Cython includes support for smart pointers, I would prefer to implement that solution. One way to do this would be a combination of malloc and a unique_ptr with a custom deleter. Another (simpler) solution would be to use the allocator class from libcpp.

import numpy as np
cimport numpy as np
from libcpp.memory cimport unique_ptr, allocator

def testArray(int dim1, int dim2):
    cdef allocator[double *] ptr_al
    cdef unique_ptr[double *] myptr
    cdef np.ndarray arr
    cdef double[:,:] carr

    myptr.reset(ptr_al.allocate(dim1))
    arr = np.ndarray((dim1,dim2),dtype=np.float64,order='C')
    carr = arr

    myptr.get()[0] = &carr[0,0]
    myptr.get()[1] = &carr[1,0]
    myptr.get()[2] = &carr[2,0]

This code compiles and executes correctly (with Cython 24.1, Python 3.5, VS2015). My concern is whether or not everything will be properly deallocated / garbage collected. My understanding is that Python is responsible for the ndarray, and unique_ptr should be responsible for the double *[] created by allocator. Is this correct, or will the code create a memory leak? Is there a way I could verify that everything has been properly deallocated?

Community
  • 1
  • 1
PaxRomana99
  • 564
  • 1
  • 5
  • 12
  • It should be noted that the allocate solution will corrupt the heap for arrays larger than 511 entries. I've created an issue for this (https://github.com/cython/cython/issues/1592). I have been able to implement the first smart pointer solution without any problems. – PaxRomana99 Jan 25 '17 at 19:43
  • Re: bug report: Switching to Py2.7 might help, if it's an option for you (I bet its support is more mature) - I tried this on Linux/GCC/Py2.7/Cy25.2 and it runs without issues for all kinds of dim1. – MWB Jan 26 '17 at 02:20
  • I tested on Linux/GCC4.8.4/Py3.5.2/Cy24.1 and it ran without issue for me as well. Maybe this is a windows thing. – PaxRomana99 Jan 27 '17 at 02:08

1 Answers1

2

Is this correct, or will the code create a memory leak?

I don't think this should leak.

Is there a way I could verify that everything has been properly deallocated?

You can call testArray in a loop and see if the process memory is growing linearly or stays about constant. Since you know dim1 and dim2, you can estimate the memory leak size if something isn't freed properly.

There are other ways to test for memory leaks, in more complicated situations: there are debug versions of the C library that will tell you if you've deallocated all of the memory you allocated. Also, there are tools like valgrind, and clang's leaksanitizer, but in your case I'd go with the loop.

MWB
  • 11,740
  • 6
  • 46
  • 91
  • I ran the loop 100 times creating a series of 100 x 100 array of doubles and the process memory remained approximately constant. Looks like it is a clean solution. – PaxRomana99 Jan 25 '17 at 17:52