Is there any way to use pythonappend with SWIG's new builtin feature?

Question

I have a little project that works beautifully with SWIG. In particular, some of my functions return std::vectors, which get translated to tuples in Python. Now, I do a lot of numerics, so I just have SWIG convert these to numpy arrays after they're returned from the c++ code. To do this, I use something like the following in SWIG.

%feature("pythonappend") My::Cool::Namespace::Data() const %{ if isinstance(val, tuple) : val = numpy.array(val) %}

(Actually, there are several functions named Data, some of which return floats, which is why I check that val is actually a tuple.) This works just beautifully.

But, I'd also like to use the -builtin flag that's now available. Calls to these Data functions are rare and mostly interactive, so their slowness is not a problem, but there are other slow loops that speed up significantly with the builtin option.

The problem is that when I use that flag, the pythonappend feature is silently ignored. Now, Data just returns a tuple again. Is there any way I could still return numpy arrays? I tried using typemaps, but it turned into a giant mess.

Edit:

Borealid has answered the question very nicely. Just for completeness, I include a couple related but subtly different typemaps that I need because I return by const reference and I use vectors of vectors (don't start!). These are different enough that I wouldn't want anyone else stumbling around trying to figure out the minor differences.

%typemap(out) std::vector<int>& {
  npy_intp result_size = $1->size();
  npy_intp dims[1] = { result_size };
  PyArrayObject* npy_arr = (PyArrayObject*)PyArray_SimpleNew(1, dims, NPY_INT);
  int* dat = (int*) PyArray_DATA(npy_arr);
  for (size_t i = 0; i < result_size; ++i) { dat[i] = (*$1)[i]; }
  $result = PyArray_Return(npy_arr);
}
%typemap(out) std::vector<std::vector<int> >& {
  npy_intp result_size = $1->size();
  npy_intp result_size2 = (result_size>0 ? (*$1)[0].size() : 0);
  npy_intp dims[2] = { result_size, result_size2 };
  PyArrayObject* npy_arr = (PyArrayObject*)PyArray_SimpleNew(2, dims, NPY_INT);
  int* dat = (int*) PyArray_DATA(npy_arr);
  for (size_t i = 0; i < result_size; ++i) { for (size_t j = 0; j < result_size2; ++j) { dat[i*result_size2+j] = (*$1)[i][j]; } }
  $result = PyArray_Return(npy_arr);
}

Edit 2:

Though not quite what I was looking for, similar problems may also be solved using @MONK's approach (explained here).

I don't think you can do this without writing a typemap and doing it on the C side, precisely because -builtin removes the code where pythonappend normally gets placed. Are you sure -builtin is much faster (i.e. did profiling lead you to use it?) I'd be tempted to use two modules, one with and one without -builtin. — Flexo, Feb 18 '12 at 11:55
I'm surprised there's no warning that `-builtin` ignores pythonappend. I am not up to the challenge of typemapping `std::vector`s into numpy arrays. I did profile, and it significantly sped up the most annoying loop in my interface (not long enough to take a break; too long to wait for frequently). But I also realized I can move this loop into my c++ code -- though somewhat awkwardly. So that's how I'll go. Still, your 'two modules' suggestion is interesting, and might be useful in other cases. — Mike, Feb 18 '12 at 14:06
Did you call SWIG with -Wall? I assumed it would warn in that case. — Flexo, Feb 18 '12 at 14:08
No warning, even with `-Wall` (though this is a big enough ignore that I don't think it should even require that). — Mike, Feb 18 '12 at 15:46

Borealid · Accepted Answer · 2013-01-11T14:10:45.497

I agree with you that using typemap gets a little messy, but it is the right way to accomplish this task. You are also right that the SWIG documentation does not directly say that %pythonappend is incompatible with -builtin, but it is strongly implied: %pythonappend adds to the Python proxy class, and the Python proxy class does not exist at all in conjunction with the -builtin flag.

Before, what you were doing was having SWIG convert the C++ std::vector objects into Python tuples, and then passing those tuples back down to numpy - where they were converted again.

What you really want to do is convert them once, at the C level.

Here's some code which will turn all std::vector<int> objects into NumPy integer arrays:

%{
#include "numpy/arrayobject.h"
%}

%init %{
    import_array();
%}

%typemap(out) std::vector<int> {
    npy_intp result_size = $1.size();

    npy_intp dims[1] = { result_size };

    PyArrayObject* npy_arr = (PyArrayObject*)PyArray_SimpleNew(1, dims, NPY_INT);
    int* dat = (int*) PyArray_DATA(npy_arr);

    for (size_t i = 0; i < result_size; ++i) {
        dat[i] = $1[i];
    }

    $result = PyArray_Return(npy_arr);
}

This uses the C-level numpy functions to construct and return an array. In order, it:

Ensures NumPy's arrayobject.h file is included in the C++ output file
Causes import_array to be called when the Python module is loaded (otherwise, all NumPy methods will segfault)
Maps any returns of std::vector<int> into NumPy arrays with a typemap

This code should be placed before you %import the headers which contain the functions returning std::vector<int>. Other than that restriction, it's entirely self-contained, so it shouldn't add too much subjective "mess" to your codebase.

If you need other vector types, you can just change the NPY_INT and all the int* and int bits, otherwise duplicating the function above.

Superb! I had all the elements you have for the typemap, but I hadn't quite put them together properly. While I don't officially have this working with my project yet, I've done a pretty thorough test by building a simpler module. Thanks very much! — Mike, Jan 11 '13 at 03:37

Is there any way to use pythonappend with SWIG's new builtin feature?

Edit:

Edit 2:

1 Answers1