Exposing STL structs in Python w/o memory leak

Question

I have a std::vector<CameraT> which I've already bound to Python thanks to Flexo's response for SWIG and C++ memory leak with vector of pointers. But I need to access fields like CameraT.t and I can't find a proper way to do it.

It is defined in a 3rd party header file (DataInterface.h) as:

template<class FT>
struct CameraT_ {
    typedef FT float_t;
    float_t t[3];
    /* more elaborate c++ stuff */
};
typedef CameraT_<float> CameraT;

The interface file looks roughly like this:

%module nvm
%{
    #define SWIG_FILE_WITH_INIT
    #include "util.h"
%}

%include "std_vector.i"

%inline %{
bool CameraDataFromNVM(const char* name, std::vector<CameraT>& camera_data){
    /* wrapper code */
}
%}

%template(CamVector) std::vector<CameraT>;

And the integration test:

import nvm

if __name__ == "__main__":
    datafile = '../../example-data/vidstills.nvm'
    cams = nvm.CamVector()

    assert nvm.CameraDataFromNVM(datafile, cams)

    print(cams[0])  # this yields no memory warnings
    print(cams[0].t[0])  # 'SwigPyObject' object has no attribute 't'

    # Yields: swig/python detected a memory leak of type 'CameraT *', no destructor found.
    c = list([i for i in cams])

prints this:

$ python3 test_nvm.py |& head
<Swig Object of type 'CameraT *' at 0x7f638228ee40>
Loading cameras/points: ../../example-data/vidstills.nvm
329 cameras; 8714 3D points; 74316 projections
Traceback (most recent call last):
  File "test_nvm.py", line 20, in <module>
    print(cams[0].t[0])
AttributeError: 'SwigPyObject' object has no attribute 't'
swig/python detected a memory leak of type 'CameraT *', no destructor found.

I've tried to place struct CameraT {} and #include statements in and outside the %inline block and failed. Also looping over the vector elements yields the memory leak warning. No idea what else to do.

The code is on Github.

Answer 1

There are a couple of problems here. First off, to fix the warning about a memory leak you need to %include "datainterface.h" and then use another %template directive, for the CameraT_ template. (The typedef doesn't change the fact that this is required).

So with:

%include "datainterface.h"
%template(CameraT) CameraT_<float>;

The warning goes away and we can access members of the type. (The lack of a warning on the first line you example you mentioned is a quirk of Python's reference counting system I think).

That's not all though, we now get an error about t not being subscriptable. We can get some insight into this by calling swig with -debug-tmsearch as an extra argument, which shows the typemaps being selected. Crucially we see something like:

datainterface.h:4: Searching for a suitable 'ret' typemap for: CameraT_< float >::float_t CameraT_< float >::t[3]
[snip lots of tries...]
Looking for: SWIGTYPE
None found

Slightly surprisingly I can't find any suitable typemaps for this in any of the SWIG headers. (It certainly exists for Java though in arrays_java.i). There are a few ways to fix it:

1. Use carrays.i and make the user do some work.
2. Switch to a container with existing wrappers. (Not an option for 3rd party code though)
3. Write some typemaps ourself.

Since 1 and 2 are trivial and not great Python I'll skip those and write a typemap or two for us, my final nvm.i file ended up looking like:

%module nvm
%{
#include "datainterface.h"
%}

%include "std_vector.i"

%typemap(out) float[ANY] %{
  $result = PyTuple_New($1_dim0);
  for (unsigned i = 0; i < $1_dim0; ++i)
    PyTuple_SetItem($result,i,PyFloat_FromDouble($1[i])); 
%}

%typemap(in) float[ANY] (unsigned i=0, float tmp[$1_dim0]) %{
  $1 = tmp;
  PyObject *item, *iterator;
  iterator = PyObject_GetIter($input); // spliting this lets a macro work

  while (!PyErr_Occurred() && iterator &&
         i < $1_dim0 && (item = PyIter_Next(iterator))) {
    $1[i++] = PyFloat_AsDouble(item);
    Py_DECREF(item);
  }

  Py_DECREF(iterator);

  if (i != $1_dim0) {
    PyErr_SetString(PyExc_AttributeError, "Failed to get $1_dim0 floats" );
    SWIG_fail;
  }
%}

%include "datainterface.h"

%inline %{
bool CameraDataFromNVM(const char* name, std::vector<CameraT>& camera_data){
  return true;
}
%}

%template(CameraT) CameraT_<float>;
%template(CamVector) std::vector<CameraT>;

Which let me run your test file with my addtions fine:

import nvm

if __name__ == "__main__":
    datafile = '../../example-data/vidstills.nvm'
    cams = nvm.CamVector(1)

    assert nvm.CameraDataFromNVM(datafile, cams)

    print(cams[0])  # this yields no memory warnings
    cams[0].t = (1,2,3)
    print(cams[0].t[0])
    #print(cams[0].bar)

    # Yields: swig/python detected a memory leak of type 'CameraT *', no destructor found.
    c = list([i for i in cams])
    print(c)

There's a caveat with this solution, which is solvable (by returning a proxy object) but not automatically a deal breaker:

cams[0].t[0] = 1 
print(cams[0].t[0]) # Won't actually have changed

This happens because we returned a new Python tuple, with a copy of t in it, so the second subscript operator here refers to that and not the original C++ data. If you wanted to fix that too, the out typemap would have to return a proxy with an implementation of __getitem__ and __setitem__ instead of just a tuple. There's a speed/complexity trade off to be had depending on how you expect to use the code. I have an example of something similar to this, you would need to adjust the out typemap to create a wrapped_array and adjust wrapped_array to hold a pointer to your real array (and a reference to the Python object so it doesn't get free'd in the wrong order).