Passing specific arguments to Boost Python function with default arguments

Question

I'm trying to create a nice interface for making HTTP calls in my game engine via Python, I have run into a bit of a problem, however.

I have a single function, get_async which launches a curl request for the specified URL. The function is defined inside an http_manager class, like so:

struct http_manager
{
    typedef function<void(boost::shared_ptr<http_response>)> response_callback_type;
    typedef function<void(size_t)> write_callback_type;

    void get_async(
        const string& url,
        const http_headers_type& headers = http_headers_type(),
        const http_data_type& data = http_data_type(),
        response_callback_type on_response = nullptr,
        write_callback_type on_write = nullptr
        );
};

I am able to successfully make this call in Python:

http.get_async('http://www.google.ca')

However, I want to make a call like this:

http.get_async('http://www.google.ca', on_response=f)

The key here is that I want to explicitly specify the arguments by name and let all the others be the defaults, just like in regular Python.

Unfortunately, when I do this, I get the following error back from Python:

ArgumentError: Python argument types in
    HttpManager.get_async(HttpManager, str)
did not match C++ signature:
    get_async(struct naga::http_manager {lvalue}, class std::basic_string<char,struct std::char_traits<char>,class std::allocator<char> > url)
    get_async(struct naga::http_manager {lvalue}, class std::basic_string<char,struct std::char_traits<char>,class std::allocator<char> > url, class std::vector<struct std::pair<class std::basic_string<char,struct std::char_traits<char>,class std::allocator<char> >,class std::basic_string<char,struct std::char_traits<char>,class std::allocator<char> > >,class std::allocator<struct std::pair<class std::basic_string<char,struct std::char_traits<char>,class std::allocator<char> >,class std::basic_string<char,struct std::char_traits<char>,class std::allocator<char> > > > > headers)
    get_async(struct naga::http_manager {lvalue}, class std::basic_string<char,struct std::char_traits<char>,class std::allocator<char> > url, class std::vector<struct std::pair<class std::basic_string<char,struct std::char_traits<char>,class std::allocator<char> >,class std::basic_string<char,struct std::char_traits<char>,class std::allocator<char> > >,class std::allocator<struct std::pair<class std::basic_string<char,struct std::char_traits<char>,class std::allocator<char> >,class std::basic_string<char,struct std::char_traits<char>,class std::allocator<char> > > > > headers, class std::map<class std::basic_string<char,struct std::char_traits<char>,class std::allocator<char> >,class std::basic_string<char,struct std::char_traits<char>,class std::allocator<char> >,struct std::less<class std::basic_string<char,struct std::char_traits<char>,class std::allocator<char> > >,class std::allocator<struct std::pair<class std::basic_string<char,struct std::char_traits<char>,class std::allocator<char> > const ,class std::basic_string<char,struct std::char_traits<char>,class std::allocator<char> > > > > data)
    get_async(struct naga::http_manager {lvalue}, class std::basic_string<char,struct std::char_traits<char>,class std::allocator<char> > url, class std::vector<struct std::pair<class std::basic_string<char,struct std::char_traits<char>,class std::allocator<char> >,class std::basic_string<char,struct std::char_traits<char>,class std::allocator<char> > >,class std::allocator<struct std::pair<class std::basic_string<char,struct std::char_traits<char>,class std::allocator<char> >,class std::basic_string<char,struct std::char_traits<char>,class std::allocator<char> > > > > headers, class std::map<class std::basic_string<char,struct std::char_traits<char>,class std::allocator<char> >,class std::basic_string<char,struct std::char_traits<char>,class std::allocator<char> >,struct std::less<class std::basic_string<char,struct std::char_traits<char>,class std::allocator<char> > >,class std::allocator<struct std::pair<class std::basic_string<char,struct std::char_traits<char>,class std::allocator<char> > const ,class std::basic_string<char,struct std::char_traits<char>,class std::allocator<char> > > > > data, class std::function<void __cdecl(class boost::shared_ptr<struct naga::http_response>)> on_response)
    get_async(struct naga::http_manager {lvalue}, class std::basic_string<char,struct std::char_traits<char>,class std::allocator<char> > url, class std::vector<struct std::pair<class std::basic_string<char,struct std::char_traits<char>,class std::allocator<char> >,class std::basic_string<char,struct std::char_traits<char>,class std::allocator<char> > >,class std::allocator<struct std::pair<class std::basic_string<char,struct std::char_traits<char>,class std::allocator<char> >,class std::basic_string<char,struct std::char_traits<char>,class std::allocator<char> > > > > headers, class std::map<class std::basic_string<char,struct std::char_traits<char>,class std::allocator<char> >,class std::basic_string<char,struct std::char_traits<char>,class std::allocator<char> >,struct std::less<class std::basic_string<char,struct std::char_traits<char>,class std::allocator<char> > >,class std::allocator<struct std::pair<class std::basic_string<char,struct std::char_traits<char>,class std::allocator<char> > const ,class std::basic_string<char,struct std::char_traits<char>,class std::allocator<char> > > > > data, class std::function<void __cdecl(class boost::shared_ptr<struct naga::http_response>)> on_response, class std::function<void __cdecl(unsigned int)> on_write)

I'm confused as to why it thinks that the argument signature is HttpManager.get_async(HttpManager, str) when I'm clearly passing 3 arguments (self, url and on_response).

Here are the relevant bits from my BOOST_PYTHON_MODULE block:

BOOST_PYTHON_MEMBER_FUNCTION_OVERLOADS(http_manager_get_async_overloads, http_manager::get_async, 1, 5)

class_<http_manager, noncopyable>("HttpManager", no_init)
    .def("get_async", &http_manager::get_async, http_manager_get_async_overloads(args("url", "headers", "data", "on_response", "on_write")))
    ;

Thank you for reading and any help is greatly appreciated!

Oh, geez--I totally missed that bottom portion of your question where you stated that you're already using Boost.Python. My bad! — villapx, Mar 14 '16 at 13:18

Tanner Sansbury · Accepted Answer · 2016-03-17T22:36:28.577

The overall problem is that C++ only uses default arguments for missing trailing arguments, and Boost.Python does not have values to provide for non-trailing missing arguments.

The Error Message

When Boost.Python internally fails to dispatch a function, the exception message only captures positional arguments. The exception message attempts to capture the caller's intentions, but display it in a C++ function call format. As C++ only supports positional arguments, there is no obvious format to displaying non-positional arguments. Consider this simple example:

#include <boost/python.hpp>

void f() {}

BOOST_PYTHON_MODULE(example)
{
  boost::python::def("f", &f);
}

Invoking example.f(a=0) throws Boost.Python.ArgumentError with the following message:

Traceback (most recent call last):
  File "<stdin>", line 2, in <module>
Boost.Python.ArgumentError: Python argument types in
    example.f()
did not match C++ signature:
    f(void)

Although Boost.Python is fully aware that the caller provided an argument, the exception only captured positional arguments. In the above example, there were no positional arguments, so the output is example.f(). In the original question, invoking http.get_async('http://www.google.ca', on_response=f) results in the exception showing HttpManager.get_async(HttpManager, str) as only two positional arguments were provided: the implicit self argument of type HttpManager and a str object.

Dispatch Failure

In C++, default arguments are not part of the function type. Furthermore, when invoking a function that has default arguments, the default arguments are only used where trailing arguments are missing. For instance, consider:

void f(int a, int b=1, int c=2);

The above function has a type of void(int, int, int), and there is no way to call function f and only provide an argument for parameter a and c. If one wishes to invoke f with a non-default value for parameter c, then all three arguments must be provided.

However, if a similar function as declared in Python, one could invoke f only providing an argument for parameter a and c as Python supports keyword arguments:

def f(a, b=1, c=2):
    pass

f(0, c=42) # Invokes f(0, b=1, c=42)

Consider this example where a C++ function with default arguments has been provided as an overloaded Python function:

#include <boost/python.hpp>

void f(int a, int b=1, int c=2) {}

BOOST_PYTHON_FUNCTION_OVERLOADS(f_overloads, f, 1, 3)

BOOST_PYTHON_MODULE(example)
{
  namespace python = boost::python;    
  python::def("f", &f, f_overloads(
    python::args("a", "b", "c")));
}

Boost.Python has been told that C++ function f has a type of void(int, int, int), and can be invoked with a minimum of 1 argument and a maximum of 3. Boost.Python is also not aware what the default arguments values are. Thus, Boost.Python creates overloads to support calling:

f(int a)
f(int a, int b)
f(int, a, int b, int c)

Hence, when one attempts to invoke example.f(0, c=42) in Python, the function dispatch will fail, as Boost.Python would need to provide a value for b when providing a value for c, and a value for b has not been specified.

Traceback (most recent call last):
  File "<stdin>", line 2, in <module>
Boost.Python.ArgumentError: Python argument types in
    example.f(int)
did not match C++ signature:
    f(int a)
    f(int a, int b)
    f(int a, int b, int c)

This behavior is why http.get_async('http://www.google.ca', on_response=f) fails, as there is no way for Boost.Python to invoke:

http_manager.get_async(url, ???, ???, on_response);

Provide Default Arguments to Python

To continue with the example, if Python was provided default arguments, it could then provide values for non-trailing arguments. When exposing a function, one can provide default values for arguments via assigning to the boost::python::arg object. The following example exposes function f with default arguments:

#include <boost/python.hpp>

void f(int a, int b=1, int c=2) {}

BOOST_PYTHON_FUNCTION_OVERLOADS(f_overloads, f, 1, 3)

BOOST_PYTHON_MODULE(example)
{
  namespace python = boost::python;    
  python::def("f", &f, f_overloads(
    (python::arg("a"), 
     python::arg("b")=1,
     python::arg("c")=2)));
}

With Boost.Python aware of default arguments, it can successfully invoke example.f(0, c=42).

Given the function signatures, argument types, and names, I recommend reading [this](http://stackoverflow.com/a/20828366/1053968) answer for details on handling multithreading with Python callbacks within Boost.Python. — Tanner Sansbury, Mar 12 '16 at 20:27
My callback system is going to be fairly simplistic. All `on_response` and `on_write` callbacks will be deferred until the start of every frame and will be called by the main thread, so I don't think I will need to deal with managing the GIL. — Colin Basnett, Mar 15 '16 at 17:22

Passing specific arguments to Boost Python function with default arguments

1 Answers1

The Error Message

Dispatch Failure

Provide Default Arguments to Python