Transferring ownership of Python object to C++ in boost-python

Question

I am trying to expose a C++ gui application to Python via boost-python and have arrived at a bit of an impasse. I am using boost-python 1.67 and Python 3.6. The target platform is Windows x64 and the compiler is VS2017.

In C++, I have an interface, lets call it Foo, and another interface, lets call it FooFactory. I also have a registry of FooFactories which contains all the FooFactories that are known about. I query this registry to find a factory and then use the factory to create Foos.

Foo and FooFactory are abstract C++ classes. In the C++ application I have a number of implementations of Foo and FooFactory already and I would like to enable users of the application to write implementations of Foo and FooFactory in Python which can then be used in the C++ application like any other Foo and FooFactory.

Here is a simplified definition of Foo and FooFactory:

#include <string>
class Foo
{
public:
  virtual ~Foo() {}
  virtual std::string getName() = 0;
  virtual void doFooThing() = 0;
};

class FooFactory
{
public:
  virtual ~FooFactory() {}
  virtual Foo* buildFoo() = 0;
};

And here is a definition of the wrappers I am using to expose the above classes to boost-python:

#include <boost/python.hpp>
#include <string>

class FooWrapper : public Foo, public boost::python::wrapper<Foo>
{
public:
  ~FooWrapper() {}
  std::string getName() override
  {
    try
    {
      if (boost::python::override func = get_override("getName"))
        return func();

        PyErr_SetString(PyExc_NotImplementedError, "'getName' unimplemented");
        boost::python::throw_error_already_set();
    }
    catch ([[maybe_unused]] boost::python::error_already_set& error)
    {
        PyErr_PrintEx(0);
    }

    return std::string();
  }

  void doFooThing() override
  {
    try
    {
      if(boost::python::override func = get_override("doFooThing"))
      {
        func();
        return;
      }

      PyErr_SetString(PyExc_NotImplementedError, "'doFooThing'");
      boost::python::throw_error_already_set();
    }
    catch([[maybe_unused]] boost::python::error_already_set& error)
    {
      PyErr_PrintEx(0);
    }
  }
};

class FooFactoryWrapper : public FooFactory, public boost::python::wrapper<FooFactory>
{
public:
  ~FooFactoryWrapper() {}

  Foo* build() override
  {
    try
    {
      if(boost::python::override func = get_override("build"))
        return func();

      PyErr_SetString(PyExc_NotImplementedError, "'build' not implemented");
      boost::python::throw_error_already_set();
    }
    catch ([[maybe_unused]] boost::python::error_already_set& error)
    {
        PyErr_PrintEx(0);
    }

    return nullptr;
  }
};

void registerFooFactory(FooFactory* factory)
{
  // fooFactoryRegistry omitted in the interest of brevity
  fooFactoryRegistry->add(factory)
}

These objects are exported to python as such:

namespace py = boost::python;
py::class_<FooWrapper, boost::noncopyable>("Foo")
  .def("getName", py::pure_virtual(&Foo::getName));

py::class_<FooFactoryWrapper, boost::noncopyable>("FooFactory")
  .def("build", py::pure_virtual(&FooFactory::build), py::return_value_policy<py::manage_new_object>());

py::def("registerFooFactory", &registerFooFactory);

And this is an example script in Python that adds a Foo and a FooFactory:

import my_module as m

class FooDerived(m.Foo):
  def getName(self):
    return "FooDerived"

class FooFactoryDerived(m.FooFactory):
  def build(self):
    return FooDerived()

m.registerFactory(FooFactoryDerived())

The registry is then used elsewhere in the application to instantiate Foos, after the termination of the Python script. Any Foo and FooFactory implementations written in Python must remain usable for the duration of the process's life.

When I run the example I have given in my C++ application, boost-python raises a

ReferenceException: Attempt to return dangling pointer to object of type: class Foo

at the line return FooDerived() in Python. I have tried numerous variations of FooFactoryWrapper and FooFactory with various shades of std::unique_ptr, std::shared_ptr, std::auto_ptr and boost::shard_ptr but none have reliably worked (either I get obscure template errors at compile time or the same result at runtime - a ReferenceException).

I was able to work around this with a stomach-churningly terrible hack where I manually call Py_INCREF() on the PyObject* inside the return value of func(), like so:

if(boost::python::override func = get_override("build"))
{
  boost::python::detail::method_result result = func();
  Py_INCREF(result.m_obj.get())
  return result;
}

but this solution won't be acceptable for production code. Even when I do this, any time in C++ when I call delete on a Python implementation of Foo, the Python implementation crashes. As an extra limitation I do not have debug symbols for the time being.

I believe that I have fundamentally misunderstood the memory ownership semantics around boost-python. As such, I have a number of questions:

1. Is this type of interface exposeable in boost-python?
2. Is it possible to create instances of Python implementations of C++ interfaces in Python whose ownership is transferred to C++?

I have looked at a number of other examples, including Boost::python: object destroys itself inside a overriden method and Boost.Python: How to expose std::unique_ptr and I have not been able to get this to work for my use case (with two abstract classes), although I am unsure if that is because I have made mistakes or have misunderstood the problem, or because it is not possible.

Any ideas?