24

I am using SWIG to interface between C++ and Python. I have created a function which creates a std::vector of object pointers. The objects that are pointed to are not important in this case.

The problem I have is that when the object (someObject) goes out of scope on the Python side it cannot free the memory pointed to by the object/s pointers within the vector, thus causing a memory leak.

Example

  • C++ code:

    std::vector < someObject* > createSomeObjectForPython()
    {
       std::vector < someObject* > myVector;
       someObject* instanceOfSomeObject = new someObject();
       myVector.push_back(instanceOfSomeObject);
       return myVector;
    }
    
  • From the Python interpreter:

    objectVar = createSomeObjectForPython()
    

When I run this in Python I get this error:

swig/python detected a memory leak of type 'std::vector< someObject *,std::allocator<  someObject * > > *', no destructor found.

This error is because when Python deletes the vector, it can only delete the pointers within the vector and not actually what they point to.

If I could create a destructor for std::vector, this would be the answer, but it’s not possible.

I really need to use vectors of pointers opposed to vectors of objects before anyone suggests this as a solution, particularly because the objects are large and complex, and speed is an issue.

I am using gcc4.4, swigwin 2.0.4, and Python 2.7 on Windows.

Alessandro Minoccheri
  • 35,521
  • 22
  • 122
  • 171
Jason
  • 285
  • 3
  • 6

1 Answers1

48

The warning you see doesn't lie directly with the fact that you have a vector of pointers. Consider the following SWIG interface file:

%module test

// This just gets passed straight through and not used for wrapping
%{
struct foo {};
%}

struct foo;

%inline %{
  struct foo bar() { struct foo f; return f; }
%}

Using this interface gives:

swig -Wall -python test.i && gcc -Wall -Wextra -std=c99 -shared -o _test.so test_wrap.c -I/usr/include/python2.7 && python2.7
Python 2.7.3 (default, Aug  1 2012, 05:16:07) 
[GCC 4.6.3] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import test
>>> test.bar()
<Swig Object of type 'struct foo *' at 0xb7654a70>
>>> 
swig/python detected a memory leak of type 'struct foo *', no destructor found.

The problem is that SWIG has only seen a declaration, not a definition for struct foo. The default behaviour is for the Python proxy object to free/delete (as appropriate) the underlying object here, but it's not able to deduce how to do that based on only the forward declaration it's seen.

If we extend the test case to include std::vector<foo> the same is observed:

%module test

%{
struct foo {};
%}

struct foo;

%include <std_vector.i>

%inline %{
  foo bar() { return foo(); }
  std::vector<foo> bar2() { 
    return std::vector<foo>(); 
  } 
%}

Which again gives the warning about no destructor:

Python 2.7.3 (default, Aug  1 2012, 05:16:07) 
[GCC 4.6.3] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import test
>>> print test.bar2()
<Swig Object of type 'std::vector< foo,std::allocator< foo > > *' at 0xb7671a70>swig/python detected a memory leak of type 'std::vector< foo,std::allocator< foo > > *', no destructor found.

However we can trivially fix this by making sure a definition of the type is available. For struct foo that's simply making the whole body of the struct visible to SWIG. For std::vector<T> we need to use %template to do that:

%module test

%include <std_vector.i>

%inline %{
  struct foo {};
  foo bar() { return foo(); }
  std::vector<foo> bar2() { 
    return std::vector<foo>(); 
  } 
%}

%template(FooVec) std::vector<foo>;

Which now doesn't warn (or leak for that matter):

Python 2.7.3 (default, Aug  1 2012, 05:16:07) 
[GCC 4.6.3] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import test
>>> print test.bar()
<test.foo; proxy of <Swig Object of type 'foo *' at 0xb76aba70> >
>>> print test.bar2()
<test.FooVec; proxy of <Swig Object of type 'std::vector< foo > *' at 0xb76abab8> >
>>> 

The complication is that in your example you have std::vector<T*>, so we can alter our test case to illustrate that:

%module test

%include <std_vector.i>

%inline %{
  struct foo {};
  foo bar() { return foo(); }
  std::vector<foo*> bar2() { 
    return std::vector<foo*>(1, new foo); 
  } 
%}

%template(FooVec) std::vector<foo*>;

Which we can then run:

Python 2.7.3 (default, Aug  1 2012, 05:16:07) 
[GCC 4.6.3] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import test
>>> print test.bar2()
<test.FooVec; proxy of <Swig Object of type 'std::vector< foo * > *' at 0xb7655a70> >
>>> 

This does leak, but crucially doesn't show the warning you noticed, because as far as SWIG is concerned the std::vector itself has been correctly deleted (the exact same semantics as in C++ in fact).

As far as how to deal with the leak there the options are the same as usual in C++. Personally I'd try to avoid putting raw pointers in a vector unless you really want the objects pointed at to outlive the vector. Basically you can:

  1. Not store pointers in the struct
  2. Use smart pointers (std::shared_ptr or std::unique_ptr or boost equivalents instead).
  3. Manage the memory manually somehow.

We've already done 1 in the second example. With SWIG 2 is pretty simple as well and 3 is a question of writing and wrapping another function in your interface.

%module test

%include <std_vector.i>
%include <std_shared_ptr.i>

%{
#include <memory>
%}

%inline %{
  struct foo {};
  foo bar() { return foo(); }
  std::vector<std::shared_ptr<foo> > bar2() { 
    return std::vector<std::shared_ptr<foo> >(1, std::make_shared<foo>()); 
  } 
%}

%shared_ptr(Foo);
%template(FooVec) std::vector<std::shared_ptr<foo> >;
Python 2.7.3 (default, Aug  1 2012, 05:16:07) 
[GCC 4.6.3] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import test
>>> print test.bar2()
<test.FooVec; proxy of <Swig Object of type 'std::vector< std::shared_ptr< foo >,std::allocator< std::shared_ptr< foo > > > *' at 0xb76f4a70> >
>>> print test.bar2()[0]
<Swig Object of type 'std::vector< std::shared_ptr< foo > >::value_type *' at 0xb76f4a70>
>>> 

Which works, stores shared pointers and doesn't leak.

If you really want to do the third way (I'd avoid it at all cost given that it leaves your interface open to human errors) the easiest way to do it with SWIG is to use %extend, for example:

%module test

%include <std_vector.i>

%inline %{
  struct foo {};
  foo bar() { return foo(); }
  std::vector<foo*> bar2() { 
    return std::vector<foo*>(1, new foo); 
  } 
%}

%template(FooVec) std::vector<foo*>;

%extend std::vector<foo*> {
  void empty_and_delete() {
    for (std::vector<foo*>::iterator it = $self->begin(); 
         it != $self->end(); ++it) {
      delete *it;
    }
    $self->clear();
  }
}

The we can do:

Python 2.7.3 (default, Aug  1 2012, 05:16:07) 
[GCC 4.6.3] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import test
>>> x = test.bar2()
>>> print x.size()
1
>>> x.empty_and_delete()
>>> print x.size()
0
>>> 

Or you could use %pythoncode to modify __del__ to call the function automatically, but that would be a bad idea because it wouldn't affect objects Python never sees at all and could lead to unexpected behaviour in a few cases.

Community
  • 1
  • 1
Flexo
  • 87,323
  • 22
  • 191
  • 272
  • 3
    Brilliant analysis. Just when I thought I understood the SWIG interface file I see that I still don't get it. Can you explain why moving %{ struct foo {}; %} to inline – Jason Nov 28 '12 at 10:09
  • 1
    Brilliant analysis....thank you for clear examples. Can you explain why (example 1) moving definition of foo from %{ struct foo {}; %} to %inline changes the behaviour? – Jason Nov 28 '12 at 10:26
  • 3
    @Jason everything inside `%{ %}` is passed through as-is to the generated wrapper file, but it's not "exposed" directly to the target language. You normally put "stuff that's needed for the generated interface to compile correctly" inside that, typically `#include` or other mechanics you want to rely upon. Stuff that's seen plain inside a SWIG interface file is wrapped, but the definition/declaration is not passed to the generated interface file directly. `%inline` is a shortcut to say "I want to wrap this *and* pass the declaration/definition through verbatim" – Flexo Nov 28 '12 at 14:11
  • 2
    So it could also be written as `%{ struct foo {}; %} struct foo {};` instead of using `%inline`. The important thing is that `struct foo {};` in a SWIG interface is not the same as `struct foo;` - the former implies a default destructor/constructor as needed, but the latter implies an opaque pointer that can only be used as a handle. – Flexo Nov 28 '12 at 14:12
  • if only all SO answers were like yours ! Thx – SAAD Aug 31 '16 at 23:55