38

I have a class interface written in C++. I have a few classes that implement this interface also written in C++. These are called in the context of a larger C++ program, which essentially implements "main". I want to be able to write implementations of this interface in Python, and allow them to be used in the context of the larger C++ program, as if they had been just written in C++.

There's been a lot written about interfacing python and C++ but I cannot quite figure out how to do what I want. The closest I can find is here: http://www.cs.brown.edu/~jwicks/boost/libs/python/doc/tutorial/doc/html/python/exposing.html#python.class_virtual_functions, but this isn't quite right.

To be more concrete, suppose I have an existing C++ interface defined something like:

// myif.h
class myif {
   public:
     virtual float myfunc(float a);
};

What I want to be able to do is something like:

// mycl.py
... some magic python stuff ...
class MyCl(myif):
  def myfunc(a):
    return a*2

Then, back in my C++ code, I want to be able to say something like:

// mymain.cc
void main(...) {
  ... some magic c++ stuff ...
  myif c = MyCl();  // get the python class
  cout << c.myfunc(5) << endl;  // should print 10
}

I hope this is sufficiently clear ;)

Flexo
  • 87,323
  • 22
  • 191
  • 272
hal3
  • 391
  • 1
  • 4
  • 4
  • 4
    "I hope this is sufficiently clear" ... C++ wrapped in python then wrapped again in C++? – AJG85 Jan 27 '12 at 22:10
  • 2
    What exactly are you hoping to accomplish by building this infrastructure? – Karl Knechtel Jan 27 '12 at 22:26
  • 3
    @AJG85, not 'wrapped in'. He asks to be able to inherit a C++ class in Python and then use that in C++. (huh!) – Johan Lundberg Jan 27 '12 at 22:29
  • 1
    This sounds like what SIP (http://www.riverbankcomputing.co.uk/software/sip/intro) is supposed to do – a lot of working with PyQT involves subclassing Qt's C++ classes. That said, I have no idea how usable it is as a standalone tool. – millimoose Jan 27 '12 at 22:37
  • @AJG85 Suppose that you have a C++ library that needs a callback object to notify the clients. This callback is implemented using a c++ abstract class, that should be realized, created and then passed to the library to be called later. If this library is going to be used in Python, you should be able to first realize the interface in Python and pass it to the C++ library. This can be one usecase. – TonySalimi Feb 11 '19 at 14:49

6 Answers6

40

There's two parts to this answer. First you need to expose your interface in Python in a way which allows Python implementations to override parts of it at will. Then you need to show your C++ program (in main how to call Python.


Exposing the existing interface to Python:

The first part is pretty easy to do with SWIG. I modified your example scenario slightly to fix a few issues and added an extra function for testing:

// myif.h
class myif {
   public:
     virtual float myfunc(float a) = 0;
};

inline void runCode(myif *inst) {
  std::cout << inst->myfunc(5) << std::endl;
}

For now I'll look at the problem without embedding Python in your application, i.e. you start excetion in Python, not in int main() in C++. It's fairly straightforward to add that later though.

First up is getting cross-language polymorphism working:

%module(directors="1") module

// We need to include myif.h in the SWIG generated C++ file
%{
#include <iostream>
#include "myif.h"
%}

// Enable cross-language polymorphism in the SWIG wrapper. 
// It's pretty slow so not enable by default
%feature("director") myif;

// Tell swig to wrap everything in myif.h
%include "myif.h"

To do that we've enabled SWIG's director feature globally and specifically for our interface. The rest of it is pretty standard SWIG though.

I wrote a test Python implementation:

import module

class MyCl(module.myif):
  def __init__(self):
    module.myif.__init__(self)
  def myfunc(self,a):
    return a*2.0

cl = MyCl()

print cl.myfunc(100.0)

module.runCode(cl)

With that I was then able to compile and run this:

swig -python  -c++ -Wall myif.i 
g++ -Wall -Wextra -shared -o _module.so myif_wrap.cxx -I/usr/include/python2.7 -lpython2.7

python mycl.py 
200.0
10

Exactly what you'd hope to see from that test.


Embedding the Python in the application:

Next up we need to implement a real version of your mymain.cc. I've put together a sketch of what it might look like:

#include <iostream>
#include "myif.h"
#include <Python.h>

int main()
{
  Py_Initialize();

  const double input = 5.0;

  PyObject *main = PyImport_AddModule("__main__");
  PyObject *dict = PyModule_GetDict(main);
  PySys_SetPath(".");
  PyObject *module = PyImport_Import(PyString_FromString("mycl"));
  PyModule_AddObject(main, "mycl", module);

  PyObject *instance = PyRun_String("mycl.MyCl()", Py_eval_input, dict, dict);
  PyObject *result = PyObject_CallMethod(instance, "myfunc", (char *)"(O)" ,PyFloat_FromDouble(input));

  PyObject *error = PyErr_Occurred();
  if (error) {
    std::cerr << "Error occured in PyRun_String" << std::endl;
    PyErr_Print();
  }

  double ret = PyFloat_AsDouble(result);
  std::cout << ret << std::endl;

  Py_Finalize();
  return 0;
}

It's basically just standard embedding Python in another application. It works and gives exactly what you'd hope to see also:

g++ -Wall -Wextra -I/usr/include/python2.7 main.cc -o main -lpython2.7
./main
200.0
10
10

The final piece of the puzzle is being able to convert the PyObject* that you get from creating the instance in Python into a myif *. SWIG again makes this reasonably straightforward.

First we need to ask SWIG to expose its runtime in a headerfile for us. We do this with an extra call to SWIG:

swig -Wall -c++ -python -external-runtime runtime.h

Next we need to re-compile our SWIG module, explicitly giving the table of types SWIG knows about a name so we can look it up from within our main.cc. We recompile the .so using:

g++ -DSWIG_TYPE_TABLE=myif -Wall -Wextra -shared -o _module.so myif_wrap.cxx -I/usr/include/python2.7 -lpython2.7

Then we add a helper function for converting the PyObject* to myif* in our main.cc:

#include "runtime.h"
// runtime.h was generated by SWIG for us with the second call we made

myif *python2interface(PyObject *obj) {
  void *argp1 = 0;
  swig_type_info * pTypeInfo = SWIG_TypeQuery("myif *");

  const int res = SWIG_ConvertPtr(obj, &argp1,pTypeInfo, 0);
  if (!SWIG_IsOK(res)) {
    abort();
  }
  return reinterpret_cast<myif*>(argp1);
}

Now this is in place we can use it from within main():

int main()
{
  Py_Initialize();

  const double input = 5.5;

  PySys_SetPath(".");
  PyObject *module = PyImport_ImportModule("mycl");

  PyObject *cls = PyObject_GetAttrString(module, "MyCl");
  PyObject *instance = PyObject_CallFunctionObjArgs(cls, NULL);

  myif *inst = python2interface(instance);
  std::cout << inst->myfunc(input) << std::endl;

  Py_XDECREF(instance);
  Py_XDECREF(cls);

  Py_Finalize();
  return 0;
}

Finally we have to compile main.cc with -DSWIG_TYPE_TABLE=myif and this gives:

./main
11
Flexo
  • 87,323
  • 22
  • 191
  • 272
  • 1
    @JonoRR - fixed. I was using the module handle rather than its dictionary as the argument to `PyRun`. – Flexo Jan 30 '12 at 10:30
  • 1
    All of the source and commands required are contained within my answer but I've also put the whole lot online at: http://static.lislan.org.uk/~ajw/pyvirt.tar.gz – Flexo Jan 30 '12 at 12:25
  • Could you comment on how the created object should be properly destructed? A quick test seems to indicate that it is necessary to run `Py_DECREF(instance)` and that simply running `delete inst` does not release the Python object. Am I correct? – mtvec Mar 03 '14 at 16:05
  • @job assuming python "owns" the underlying native object Py_DECREF is necessary and sufficient to delete it. If you delete the interface and python owns it you'll end up with a double delete if you don't also revoke ownership. – Flexo Mar 03 '14 at 17:24
  • Thanks for the explanation! For those who, like me, want C++ to have ownership of the object and be able to delete objects the C++ way, here is what you have to do: 1) create the instance by running `mycl.MyCl().__disown__()`, 2) cast the instance to your interface by calling `python2interface`, 3) run `Py_DECREF` on the `PyObject` instance, 4) when you're done with your object, destroy it using `delete`, this should also destroy the Python object (assuming there are no more references to it in Python, which there shouldn't be if you created it using Flexo's method). – mtvec Mar 04 '14 at 09:54
  • Brilliant solution. On Windows using VStudio, I had to create two separate projects, one to produce _module.pyd (which is a rename from _module.dll, but Python searches for the *.pyd), and the actual Main which is the *.exe entry point. Thank you. – Mihai Galos Aug 26 '16 at 22:25
12

Minimal example; note that it is complicated by the fact that Base is not pure virtual. There we go:

  1. baz.cpp:

    #include<string>
    #include<boost/python.hpp>
    using std::string;
    namespace py=boost::python;
    
    struct Base{
      virtual string foo() const { return "Base.foo"; }
      // fooBase is non-virtual, calling it from anywhere (c++ or python)
      // will go through c++ dispatch
      string fooBase() const { return foo(); }
    };
    struct BaseWrapper: Base, py::wrapper<Base>{
      string foo() const{
        // if Base were abstract (non-instantiable in python), then
        // there would be only this->get_override("foo")() here
        //
        // if called on a class which overrides foo in python
        if(this->get_override("foo")) return this->get_override("foo")();
        // no override in python; happens if Base(Wrapper) is instantiated directly
        else return Base::foo();
      }
    };
    
    BOOST_PYTHON_MODULE(baz){
      py::class_<BaseWrapper,boost::noncopyable>("Base")
        .def("foo",&Base::foo)
        .def("fooBase",&Base::fooBase)
      ;
    }
    
  2. bar.py

    import sys
    sys.path.append('.')
    import baz
    
    class PyDerived(baz.Base):
      def foo(self): return 'PyDerived.foo'
    
    base=baz.Base()
    der=PyDerived()
    print base.foo(), base.fooBase()
    print der.foo(), der.fooBase()
    
  3. Makefile

    default:
           g++ -shared -fPIC -o baz.so baz.cpp -lboost_python `pkg-config python --cflags`
    

And the result is:

Base.foo Base.foo
PyDerived.foo PyDerived.foo

where you can see how fooBase() (the non-virtual c++ function) calls virtual foo(), which resolves to the override regardless whether in c++ or python. You could derive a class from Base in c++ and it would work just the same.

EDIT (extracting c++ object):

PyObject* obj; // given
py::object pyObj(obj); // wrap as boost::python object (cheap)
py::extract<Base> ex(pyObj); 
if(ex.check()){ // types are compatible
  Base& b=ex(); // get the wrapped object
  // ...
} else {
  // error
}

// shorter, thrwos when conversion not possible
Base &b=py::extract<Base>(py::object(obj))();

Construct py::object from PyObject* and use py::extract to query whether the python object matches what you are trying to extract: PyObject* obj; py::extract<Base> extractor(py::object(obj)); if(!extractor.check()) /* error */; Base& b=extractor();

eudoxos
  • 18,545
  • 10
  • 61
  • 110
  • This looks pretty good - can you add the equivalent Boost.Python embedding the interpreter within `main()` to the one in my example which uses the `myif *python2interface(PyObject *obj);` to take a PyObject and convert it back to an implementation of the C++ interface? (With that it'd be exactly the answer I had in mind for the bounty) – Flexo Jan 31 '12 at 10:58
  • I've awarded you the bounty for this, thanks! I've also extended your solution to both embed and extend simultaneously, which I've posted as a second answer. – Flexo Feb 01 '12 at 18:16
10

Quoting http://wiki.python.org/moin/boost.python/Inheritance

"Boost.Python also allows us to represent C++ inheritance relationships so that wrapped derived classes may be passed where values, pointers, or references to a base class are expected as arguments."

There are examples of virtual functions so that solves the first part (the one with class MyCl(myif))

For specific examples doing this, http://wiki.python.org/moin/boost.python/OverridableVirtualFunctions

For the line myif c = MyCl(); you need to expose your python (module) to C++. There are examples here http://wiki.python.org/moin/boost.python/EmbeddingPython

Johan Lundberg
  • 26,184
  • 12
  • 71
  • 97
  • Can you expand this into a complete example? I've put a bounty on it if that inspires you :) – Flexo Jan 30 '12 at 17:15
  • I wasn't the downvoter, but I suspect the lack of detail and slightly speculative nature of this answer was behind it. That is certainly the reason I offered the bounty on it. – Flexo Jan 31 '12 at 11:33
8

Based upon the (very helpful) answer by Eudoxos I've taken his code and extended it such that there is now an embedded interpreter, with a built-in module.

This answer is the Boost.Python equivalent of my SWIG based answer.

The headerfile myif.h:

class myif {
public:
  virtual float myfunc(float a) const { return 0; }
  virtual ~myif() {}
};

Is basically as in the question, but with a default implementation of myfunc and a virtual destructor.

For the Python implementation, MyCl.py I have basically the same as the question:

import myif

class MyCl(myif.myif):
  def myfunc(self,a): 
    return a*2.0

This then leaves mymain.cc, most of which is based upon the answer from Eudoxos:

#include <boost/python.hpp>
#include <iostream>
#include "myif.h"

using namespace boost::python;

// This is basically Eudoxos's answer:
struct MyIfWrapper: myif, wrapper<myif>{
  float myfunc(float a) const {
    if(this->get_override("myfunc")) 
      return this->get_override("myfunc")(a);
    else 
      return myif::myfunc(a);
  }
};

BOOST_PYTHON_MODULE(myif){
  class_<MyIfWrapper,boost::noncopyable>("myif")
    .def("myfunc",&myif::myfunc)
  ;
}
// End answer by Eudoxos

int main( int argc, char ** argv ) {
  try {
    // Tell python that "myif" is a built-in module
    PyImport_AppendInittab("myif", initmyif);
    // Set up embedded Python interpreter:
    Py_Initialize();

    object main_module = import("__main__");
    object main_namespace = main_module.attr("__dict__");

    PySys_SetPath(".");
    main_namespace["mycl"] = import("mycl");

    // Create the Python object with an eval()
    object obj = eval("mycl.MyCl()", main_namespace);

    // Find the base C++ type for the Python object (from Eudoxos)
    const myif &b=extract<myif>(obj)();
    std::cout << b.myfunc(5) << std::endl;

  } catch( error_already_set ) {
    PyErr_Print();
  }
}

The key part that I've added here, above and beyond the "how do I embed Python using Boost.Python?" and "how do I extend Python using Boost.python?" (which was answered by Eudoxos) is the answer to the question "How do I do both at once in the same program?". The solution to this lies with the PyImport_AppendInittab call, which takes the initialisation function that would normally be called when the module is loaded and registers it as a built-in module. Thus when mycl.py says import myif it ends up importing the built-in Boost.Python module.

Community
  • 1
  • 1
Flexo
  • 87,323
  • 22
  • 191
  • 272
1

Take a look at Boost Python, that is the most versatile and powerful tool to bridge between C++ and Python.

http://www.boost.org/doc/libs/1_48_0/libs/python/doc/

pyroscope
  • 4,120
  • 1
  • 18
  • 13
  • 1
    It's only as powerful as the Python C API, but it's definitively a great library. [One thing that I'm looking for](http://stackoverflow.com/q/9050985/627005) and you apparently can't do with Boost is to define Python metaclasses in C++; you can do that with the C API, though. – Paul Manta Jan 30 '12 at 18:53
-2

There's no real way to interface C++ code directly with Python.

SWIG does handle this, but it builds its own wrapper.

One alternative I prefer over SWIG is ctypes, but to use this you need to create a C wrapper.

For the example:

// myif.h
class myif {
   public:
     virtual float myfunc(float a);
};

Build a C wrapper like so:

extern "C" __declspec(dllexport) float myif_myfunc(myif* m, float a) {
    return m->myfunc(a);
}

Since you are building using C++, the extern "C" allows for C linkage so you can call it easily from your dll, and __declspec(dllexport) allows the function to be called from the dll.

In Python:

from ctypes import *
from os.path import dirname

dlldir = dirname(__file__)                      # this strips it to the directory only
dlldir.replace( '\\', '\\\\' )                  # Replaces \ with \\ in dlldir
lib = cdll.LoadLibrary(dlldir+'\\myif.dll')     # Loads from the full path to your module.

# Just an alias for the void pointer for your class
c_myif = c_void_p

# This tells Python how to interpret the return type and arguments
lib.myif_myfunc.argtypes = [ c_myif, c_float ]
lib.myif_myfunc.restype  = c_float

class MyCl(myif):
    def __init__:
        # Assume you wrapped a constructor for myif in C
        self.obj = lib.myif_newmyif(None)

    def myfunc(a):
        return lib.myif_myfunc(self.obj, a)

While SWIG does all this for you, there's little room for you to modify things as you please without getting frustrated at all the changes you have to redo when you regenerate the SWIG wrapper.

One issue with ctypes is that it doesn't handle STL structures, since it's made for C. SWIG does handle this for you, but you may be able to wrap it yourself in the C. It's up to you.

Here's the Python doc for ctypes:

http://docs.python.org/library/ctypes.html

Also, the built dll should be in the same folder as your Python interface (why wouldn't it be?).

I am curious though, why would you want to call Python from inside C++ instead of calling the C++ implementation directly?

Pochi
  • 2,026
  • 1
  • 15
  • 11
  • 1
    *"there's little room for you to modify things as you please without getting frustrated at all the changes you have to redo when you regenerate the SWIG wrapper"* - you should *never* edit a SWIG generated file by hand. There's always a way of making SWIG generate the code you want. – Flexo Jan 30 '12 at 18:59
  • In terms of the bounty this answer doesn't add much that I can see - This doesn't solve the embedding and "how do I pass an instance of my Python type that derives from the base interface back to a C++ function problem either". I could write a proxy that handled that, but I was looking for a solution that involved writing *less* glue or neater glue than the SWIG one I proposed. This seems to add more manually written glue and not address the embedding Python in C++ issue. – Flexo Jan 30 '12 at 19:03