Interoperability between C++ Boost Multiprecision and Python's mpmath

Question

I have experience working with both Boost Multiprecision and with Python's mpmath, separately.

When it gets to making both communicate (for example to create Python extensions in C++), my attempts have always involved some sort of wasteful float-to-string and string-to-float conversion.

My question is: is it possible to make both communicate in a more performant (and elegant) way? And by that I mean, is there a way to directly have C++ Boost Multiprecision load from and export to a Python mpmath.mpf object in the same vein as C's mpp does via pybind11?

I have been searching for this for quite a bit. The only other similar question I found was about just exporting from Boost Multiprecision to Python (in general) using pybind11, not to a mpmath object directly. And in that question, the OP ended up using the same approach I am trying to avoid (that is, converting from/to strings when communicating from/to C++ and Python).

Answer 1

This answers only partially to your question. Because the direct answer is: No, it is not possible in a clean way without a wasteful conversion to string, because mpmath is a purely python library without any parts of it written in C or C++, hence even if you try to skip "wasteful conversion" by seeking to use some sort of binary compatibility, your code will be very fragile: it will break every time when some python or mpmath internals are changed ever so slightly.

However I needed exactly the same thing. And so I settled down for an automated conversion registered via boost::python which checks and converts using strings. Actually inside python you also create mpmath.mpf objects from strings, so it's very much the same, except in the code below it is faster because it is written inside C++.

So here's what works for me:

#include <boost/python.hpp>
#include <iostream>
#include <limits>
#include <sstream>
#include <boost/math/constants/constants.hpp>
#include <boost/multiprecision/cpp_bin_float.hpp>
namespace py = ::boost::python;
using Prec80 = boost::multiprecision::number<boost::multiprecision::cpp_bin_float<80>>;

template<typename ArbitraryReal>
struct ArbitraryReal_to_python {
    static PyObject* convert(const ArbitraryReal& val){
        std::stringstream ss{};
        ss << std::setprecision(std::numeric_limits<ArbitraryReal>::digits10+1) << val;
        py::object mpmath = py::import("mpmath");
        mpmath.attr("mp").attr("dps")=int(std::numeric_limits<ArbitraryReal>::digits10+1);
        py::object result = mpmath.attr("mpf")(ss.str());
        return boost::python::incref( result.ptr() );
    }
};

template<typename ArbitraryReal>
struct ArbitraryReal_from_python {
    ArbitraryReal_from_python(){
         boost::python::converter::registry::push_back(&convertible,&construct,boost::python::type_id<ArbitraryReal>());
    }
    static void* convertible(PyObject* obj_ptr){
        // Accept whatever python is able to convert into float
        // This works with mpmath numbers. However if you want to accept strings as numbers this checking code can be a little longer to verify if string is a valid number.
        double check = PyFloat_AsDouble(obj_ptr);
        return (PyErr_Occurred()==nullptr) ? obj_ptr : nullptr;
    }
    static void construct(PyObject* obj_ptr, boost::python::converter::rvalue_from_python_stage1_data* data){
        std::istringstream ss{ py::call_method<std::string>(obj_ptr, "__str__") };
        void* storage=((boost::python::converter::rvalue_from_python_storage<ArbitraryReal>*)(data))->storage.bytes;
        new (storage) ArbitraryReal;
        ArbitraryReal* val=(ArbitraryReal*)storage;
        ss >> *val;
        data->convertible=storage;
    }
};


struct Var
{
    Prec80 value{"-71.23"};
    Prec80 get() const   { return value; };
    void set(Prec80 val) { value = val;  };
};

BOOST_PYTHON_MODULE(pysmall)
{
    ArbitraryReal_from_python<Prec80>();
    py::to_python_converter<Prec80,ArbitraryReal_to_python<Prec80>>();

    py::class_<Var>("Var" )
        .add_property("val", &Var::get, &Var::set);
}

Now you compile this code with this command:

g++ -O1 -g pysmall.cpp -o pysmall.so -std=gnu++17 -fPIC -shared -I/usr/include/python3.7m/ -lboost_python37 -lpython3.7m -Wl,-soname,"pysmall.so"

And here is an example python session:

In [1]: import pysmall
In [2]: a=pysmall.Var()
In [3]: a.val
Out[3]: mpf('-71.2299999999999999999999999999999999999999999999999999999999999999999999999999997072')
In [4]: a.val=123.12
In [5]: a.val
Out[5]: mpf('123.120000000000000000000000000000000000000000000000000000000000000000000000000000003')

The C++ code does not care whether mpmath is already imported in python. If it is, it obtains the exsiting library handle, if it is not then it imports it. If you find any room for improvement in this snippet please let me know!

Here's a couple of useful references when I was writing this:

1. https://misspent.wordpress.com/2009/09/27/how-to-write-boost-python-converters/
2. https://github.com/bluescarni/mppp/blob/master/include/mp%2B%2B/extra/pybind11.hpp (but I didn't want to use pybind11, just boost::python)

EDIT: I have now finished implementing this in YADE , it works with EIGEN and CGAL libraries. The part concerning this question is in file ToFromPythonConverter.hpp