Pass a variable number of parameters to an embedded python API

Question

Today I was reading about embedded python in C++ at

https://docs.python.org/3/extending/embedding.html

So, I can call a python code in C++.

But the way python is called in the API example does not look cool to me.

I was thinking about calling a python function from C++ in any arbitrary way like:

py_call(script_path,module_name,str1,int2,long3,float4,str5,double6);

or

py_call(script_path,module_name,x,y,z,title);

But I need to use a parameter pack. This is the first time I see parameter pack. I have stuck right here and I do not know how to replace argc and argv parameters in the following code:

template<typename T, typename... Targs>
void py_call(
        const string &script,
        const string &module,
        T value, Targs... Fargs
    )
{
    PyObject *pName, *pModule, *pFunc;
    PyObject *pArgs, *pValue;
    int i;

    pName = PyUnicode_DecodeFSDefault(script.c_str());
    /* Error checking of pName left out */

    pModule = PyImport_Import(pName);
    Py_DECREF(pName);
    if (pModule != NULL) {
        pFunc = PyObject_GetAttrString(pModule, module.c_str());
        /* pFunc is a new reference */

        if (pFunc && PyCallable_Check(pFunc)) {
            pArgs = PyTuple_New(argc - 3);
            for (i = 0; i < argc - 3; ++i) {
                pValue = PyLong_FromLong(atoi(argv[i + 3]));
                if (!pValue) {
                    Py_DECREF(pArgs);
                    Py_DECREF(pModule);
                    fprintf(stderr, "Cannot convert argument\n");
                    return 1;
                }
                /* pValue reference stolen here: */
                PyTuple_SetItem(pArgs, i, pValue);
            }
            pValue = PyObject_CallObject(pFunc, pArgs);
            Py_DECREF(pArgs);
            if (pValue != NULL) {
                printf("Result of call: %ld\n", PyLong_AsLong(pValue));
                Py_DECREF(pValue);
            }
            else {
                Py_DECREF(pFunc);
                Py_DECREF(pModule);
                PyErr_Print();
                fprintf(stderr,"Call failed\n");
                return 1;
            }
        }
        else
        {
            if (PyErr_Occurred())
                PyErr_Print();
            fprintf(stderr, "Cannot find function \"%s\"\n", module.c_str());
        }
        Py_XDECREF(pFunc);
        Py_DECREF(pModule);
    }
    else
    {
        PyErr_Print();
        fprintf(stderr, "Failed to load \"%s\"\n",script.c_str());
        return 1;
    }
}

PS. argc=sizeof...(Fargs)+1 or argc=sizeof...(Fargs) depending on the function implementation.

Your `py_call` uses a mixture of types, whereas in your example, all arguments are converted to `int`. The `argv` from where you have taken some parts, are all `char*`. If you want a single solution to cover all signatures, you need to convert all types to PyObjects - not just integers. I can recommend you to look into SWIG — Jens Munk, Aug 04 '18 at 08:29
@JensMunk Is it possible that I write a user defined type-case? — ar2015, Aug 04 '18 at 08:31
Even better, you could use template function overloads, such that if the type is integer, `PyLong_FromLong` is used and another function if the type is double. See e.g. https://stackoverflow.com/questions/2174300/function-template-overloading — Jens Munk, Aug 04 '18 at 08:35
The author of FluentCPP explains it quite well, https://www.fluentcpp.com/2017/08/15/function-templates-partial-specialization-cpp/ — Jens Munk, Aug 04 '18 at 08:42

score 1 · Accepted Answer · answered Aug 04 '18 at 10:04

I'd advice splitting construction of argv tuple from the rest of your code. In order to construct this tuple, you need to iterate over the parameter pack and for each parameter create a PyObject by means of overloading. C++11 version could look somehow like this:

void to_py_tuple_impl(PyObject*, size_t) {}

template<typename ParamType, typename... ParamTypesTail>
void to_py_tuple_impl(PyObject* tpl, size_t index, const ParamType& param, const ParamTypesTail&... tail)
{
    // error checking omitted for clarity
    PyTuple_SetItem(tpl, index, to_py_object(param));
    to_py_tuple_impl(tpl, index + 1, tail...);
}

template<typename... ParamTypes>
PyObject* to_py_tuple(const ParamTypes&... args)
{
    PyObject* tpl = PyTuple_New(sizeof...(ParamTypes));
    to_py_tuple_impl(tpl, 0, args...);
    return tpl;
}

If you have to stick with C++11, you need to use a recursive function to iterate over args - that's what to_py_tuple_impl does. I've wrapped it into to_py_tuple for simplicity.

This code calls to_py_object for each argument in order to convert it to PyObject* and inserts that object into a tuple.

to_py_object can be overloaded to support multiple types, for example:

PyObject* to_py_object(const std::string& str)
{
    return PyUnicode_FromStringAndSize(str.c_str(), str.size());
}

PyObject* to_py_object(const char* str)
{
    retrurn PyUnicode_FromString(str);
}

Number of overloads can be reduced using templates and std::enable_if:

// Converts all integer types:
template<typename T>
std::enable_if_t<std::is_integral<T>::value, PyObject*> to_py_object(T value)
{
    return PyLong_FromLong(value);
}

// Converts all floating point types:
template<typename T>
std::enable_if_t<std::is_floating_point<T>::value, PyObject*> to_py_object(T value)
{
    return PyFloat_FromDouble(value);
}

Now, you can just plug it into your py_call function:

template<typename T, typename... Targs>
void py_call(
        const string &script,
        const string &module,
        T value, Targs... Fargs
    )
{
    PyObject *pName, *pModule, *pFunc;
    PyObject *pArgs, *pValue;
    int i;

    pName = PyUnicode_DecodeFSDefault(script.c_str());
    /* Error checking of pName left out */

    pModule = PyImport_Import(pName);
    Py_DECREF(pName);
    if (pModule != NULL) {
        pFunc = PyObject_GetAttrString(pModule, module.c_str());
        /* pFunc is a new reference */

        if (pFunc && PyCallable_Check(pFunc)) {
            pArgs = to_py_tuple(Fargs...);
            pValue = PyObject_CallObject(pFunc, pArgs);

            ...

Pass a variable number of parameters to an embedded python API

1 Answers1