I am trying to call a python function from a C++ code which contains main() function using Pybind11. But I found very few references are available. Most of existing documents talk about the reversed direction, i.e. calling C++ from Python.
Is there any complete example showing how to do that? The only reference I found is: https://github.com/pybind/pybind11/issues/30
But it has very little information.
The answer to your question really has two parts: one about calling a Python function from C++, the other about embedding the interpreter.
Calling a function in pybind11 is simply a matter of getting that function into a pybind11::object variable, on which you can invoke operator() to attempt to call the object. (It doesn't have to be a function, but just something callable: for example, it could also be an object with a __call__ method). For example, to call math.sqrt(2) from C++ code you'd use:
auto math = py::module::import("math");
auto resultobj = math.attr("sqrt")(2);
double result = resultobj.cast<double>();
or you could condense it all to just:
double result = py::module::import("math").attr("sqrt")(2).cast<double>();
The second part of the question involves how to do this from a C++ executable. When building an executable (i.e. when your C++ code contains main()) you have to embed the Python interpreter in your binary before you can do anything with Python (like calling a Python function).
Embedded support is a new feature added in the current pybind11 master branch (which will become the 2.2 release). Here's a basic example that starts an embedded Python interpreter and calls a Python function (math.sqrt):
#include <pybind11/embed.h>
#include <iostream>
namespace py = pybind11;
int main() {
py::scoped_interpreter python;
auto math = py::module::import("math");
double root_two = math.attr("sqrt")(2.0).cast<double>();
std::cout << "The square root of 2 is: " << root_two << "\n";
}
Outputs:
The square root of 2 is: 1.41421
More examples and documentation of calling functions and embedding are available at http://pybind11.readthedocs.io/en/master/advanced/pycpp/object.html and http://pybind11.readthedocs.io/en/master/advanced/embedding.html, respectively.
Jasons answer is pretty much on point, but I want to add a slightly more complex (and clean) example calling a python method with a numpy input.
I want to showcase two points:
py::object to a py::function using py::reinterpret_borrow<py::function>std::vector that automatically gets converted to a numpy.arrayNote that the user is responsible for making sure that the PyModule.attr is actually a python function. Also note that the type conversion works for a wide variety of c++ types (see here for details).
In this example I want to use the method scipy.optimize.minimize with a starting point x0 that is provided from the c++ interface.
#include <iostream>
#include <vector>
#include <pybind11/pybind11.h>
#include <pybind11/embed.h> // python interpreter
#include <pybind11/stl.h> // type conversion
namespace py = pybind11;
int main() {
std::cout << "Starting pybind" << std::endl;
py::scoped_interpreter guard{}; // start interpreter, dies when out of scope
py::function min_rosen =
py::reinterpret_borrow<py::function>( // cast from 'object' to 'function - use `borrow` (copy) or `steal` (move)
py::module::import("py_src.exec_numpy").attr("min_rosen") // import method "min_rosen" from python "module"
);
py::object result = min_rosen(std::vector<double>{1,2,3,4,5}); // automatic conversion from `std::vector` to `numpy.array`, imported in `pybind11/stl.h`
bool success = result.attr("success").cast<bool>();
int num_iters = result.attr("nit").cast<int>();
double obj_value = result.attr("fun").cast<double>();
}
with the python script py_src/exec_numpy.py
import numpy as np
from scipy.optimize import minimize, rosen, rosen_der
def min_rosen(x0):
res = minimize(rosen, x0)
return res
Hope this helps someone!
main.cpp
#include <pybind11/embed.h>
#include <iostream>
namespace py = pybind11;
using namespace py::literals;
int main() {
py::scoped_interpreter guard{};
// append source dir to sys.path, and python interpreter would find your custom python file
py::module_ sys = py::module_::import("sys");
py::list path = sys.attr("path");
path.attr("append")("..");
// import custom python class and call it
py::module_ tokenize = py::module_::import("calc");
py::type customTokenizerClass = tokenize.attr("CustomTokenizer");
py::object customTokenizer = customTokenizerClass("/Users/Caleb/Desktop/codes/ptms/bert-base");
py::object res = customTokenizer.attr("custom_tokenize")("good luck");
// show the result
py::list input_ids = res.attr("input_ids");
py::list token_type_ids = res.attr("token_type_ids");
py::list attention_mask = res.attr("attention_mask");
py::list offsets = res.attr("offset_mapping");
std::string message = "input ids is {},\noffsets is {}"_s.format(input_ids, offsets);
std::cout << message << std::endl;
}
calc.py
from transformers import BertTokenizerFast
class CustomTokenizer(object):
def __init__(self, vocab_dir):
self._tokenizer = BertTokenizerFast.from_pretrained(vocab_dir)
def custom_tokenize(self, text):
return self._tokenizer(text, return_offsets_mapping=True)
def build_tokenizer(vocab_dir: str) -> BertTokenizerFast:
tokenizer = BertTokenizerFast.from_pretrained(vocab_dir)
return tokenizer
def tokenize_text(tokenizer: BertTokenizerFast, text: str) -> dict:
res = tokenizer(text, return_offsets_mapping=True)
return dict(res)
CMakeLists.txt
cmake_minimum_required(VERSION 3.4)
project(example)
set(CMAKE_CXX_STANDARD 11)
# set pybind11 dir
set(pybind11_DIR /Users/Caleb/Softwares/pybind11)
find_package(pybind11 REQUIRED)
# set custom python interpreter(under macos)
link_libraries(/Users/Caleb/miniforge3/envs/py38/lib/libpython3.8.dylib)
add_executable(example main.cpp)
target_link_libraries(example PRIVATE pybind11::embed)
