Difficulties using Variadic Templates

Question

I'm writing a networking-related class. My application receives network messages of the form

[uint8_t message id, uint8_t/uint16_t/uint32_t data ...]

My class allows its user to register a callback for a specific message id.

Since there are variety of different messages with different number of different data entries (data entries are restricted to uint8_t, uint16_t and uint32_t), I decided to use C++11's variadic templates to lessen the burden of repeated code.

Here is my pseudo-code of what I want to do (didn't compile it and doubt it compiles)

#include <arpa/inet.h>
#include <stdexcept>

using namespace std;

template<class ...T>
struct MessageHandler {
    size_t size;
    std::function<void(T...)> callback;

    template<class Head, class... Tail>
    void parseHelper(uint8_t *data)
    {
        if (sizeof(Head) == 1) {
            uint8_t val;
            memcpy(&val, data, sizeof(Head));
            // set next unset argument to the value of val
            callback = std::bind(callback, val);
            data += sizeof(Head);
        } else if (sizeof(Head) == 2) {
            uint16_t val;
            memcpy(&val, data, sizeof(Head));
            val = ntohs(val);
            // set next unset argument to the value of val
            callback = std::bind(callback, val);
            data += sizeof(Head);
        } else if (sizeof(Head) == 4) {
            uint32_t val;
            memcpy(&val, data, sizeof(Head));
            val = ntohl(val);
            // set next unset argument to the value of val
            callback = std::bind(callback, val);
            data += sizeof(Head);
        } else {
            throw std::invalid_argument("We support only 1, 2 and 4 byte integers!");
        }

        // repeat for the rest of arguments
        parseHelper<Tail...>(data);
    }

    template<class ...Empty>
    void parseHelper(uint8_t *data)
    {
        // do nothing, terminating case of recursion
    }

    template<class ...T>
    void parse(utin8_t *data)
    {
        // parse `data` into T... arguments and bind them into `callback`
        parseHelper<T...>(data);

        // at this point `callback` has all arguments binded from `data`

        // invoke the callback
        callback();
    }
}

// <message id, callback-holding helper struct>
std::unordered_map<uint8_t, MessageHandler> myMap;

template<class...T>
void dummy(T&&...)
{
    // a dummy, does nothing
}

template<class...T>
void addMessageHandler(uint8_t messageId, std::function<void<T... arg>> callback)
{
    MessageHandler<arg> mh;

    mh.size = 0;
    // order of execution is undefined, but we don't care
    dummy( (mh.size += sizeof(arg))... );

    mh.callback = callback;

    myMap[messageId] = mh;
}

void foo(uint16_t a, uint8_t b, uint16_t c, uint32_t d)
{
    // do stuff with the parsed message
}

void bar(uint32_t a)
{
    // do stuff with the parsed message
}

int main()
{
    // register callbacks
    addMessageHandler<uint16_t, uint8_t, uint16_t, uint32_t>(0, std::bind(&foo));
    addMessageHandler<uint32_t>(1, std::bind(&bar));
    ...

    // get message over the network
    uint8_t messageId = some_network_library.read.first_byte();
    MessageHandler mh = myMap[messageId];
    uint8_t *data = some_network_library.read.bytes(mh.size);
    // parses and calls the callback with parsed values
    mh.parse(data);

    return 0;
}

In main, we register callbacks for message ids, then receive a message over the network, get appropriate MessageHandler, parse data variable by variable, appending each of them to the callback bind, and when we have binded everything — call the callback.

So, things that concern me:

1. Is it even possible to have a map (or some other integer-key struct-value based data-structure with approximately constant lookup) where the value is a template struct and you want to store structs of different type in it? (i.e. values stored in the map are not of homogeneous type).

2. What do I need to make parse and parseHelper functions to work?

   • I'm not sure if you can append-bind values to std::function like that
   • After calling the callback in parse, how do I unbind all the bind values? (or they automatically unbind after the call?)

How do I make this code work?

It would be great if someone could fix my pseudo-code into a working one, explaining why my code wouldn't work and how it's fixable, but just explanations are very very helpful too!

Answer 1

1. parametric polymorphism (templates) is not inclusion polymorphism (inheritance): MessageHandler<int> and MessageHandler<float> are different types and don't share a common definition that can be used for the other (a "base" class). So no, you cannot create a container that can store MessageHandler with different parameters.

Keep also in mind that static typing also implies to know the size of declaration. Which is not possible without solving the parameters to their actual "values".

So no. You cannot have a map<key, MessageHandler<T...>> without actually specifying T, which forbids using multiple values for T....

To solve this problem, you can use a type eraser. We use this for example:

https://github.com/aerys/minko/blob/master/framework/include/minko/Any.hpp

so we can create a map<key, Any>.

2. If you want to have a variadic callback approach, you could have a look at our Signal class:

https://github.com/aerys/minko/blob/master/framework/include/minko/Signal.hpp

It uses variadic templates to call callbacks with the corresponding parameters as arguments.

In the case of your parseHelper function, I think it has multiple issues:

• It will take only of the "head" value, don't you need some kind of loop/recursive call?
• If you do such loop, when should it stop? You need both the pointer and the size of the "message" you're reading.
• I think you should use lambdas instead of std::bind: everything is monomorphic in your case, so std::bind will take a lot more memory/CPU for nothing.
• Don't you want to call callback instead of setting it? I thought callback was a user defined value?

I think what you want to do is to "deserialize" the set of values coming from the network and then pass those values as arguments of your callback. Is that correct?

If that's the case you can have a look at this: https://stackoverflow.com/a/1547118/4525791

Answer 2

You could easily parse dynamic data from the memory by template arguments (see part 1). On how to call a function with the tuple the answer is very helpful and can be applied (see part 2). Now all you need is to store the information about function and call with dynamically parsed values. Also from my point of view there can be another problem with finding size of message when reading it, so I made simple helper struct for it

template< typename T1, typename... T2 >
struct size_of {
    enum {
        size = sizeof (T1) + size_of < T2... >::size
    };
};

template< typename T >
struct size_of< T > {
    enum {
        size = sizeof (T)
    };
};

So here is the code with some comments

1. Parser

   template< typename T1, typename... T2 >
   struct parser {
       static std::tuple< T1, T2... > parse(void* data) {
           // get value from pointer
           T1* p = (T1*) data;
           std::cout << typeid (*p).name() << " " << *p << std::endl;
           // concatenate current value with next one 
           return std::tuple_cat(std::make_tuple(*p),
                   parser < T2... >::parse(p + 1));
       }
   };
   
   template< typename T1 >
   struct parser< T1 > {
       static std::tuple< T1 > parse(void* data) {
           T1* p = (T1*) data;
           std::cout << typeid (*p).name() << " " << *p << std::endl;
           return std::make_tuple(*p);
       }
   };
   

   Also you can reimplement this class to return size of parsed values to be sure everything is ok.

2. Function call

   // function call using tuple 
   template < int N >
   struct __apply_impl {
       template < typename... ArgsF, typename... ArgsT, typename... Args >
       static void apply(const std::function<void( ArgsF...)>& f,
               const std::tuple<ArgsT...>& t,
               Args... args) {
           __apply_impl < N - 1 > ::apply(f, t, std::get < N - 1 > (t), args...);
       }
   };
   
   template <>
   struct __apply_impl<0> {
       template < typename... ArgsF, typename... ArgsT, typename... Args >
       static void apply(const std::function<void( ArgsF...)>& f,
               const std::tuple<ArgsT...>& /* t */,
               Args... args) {
           // actual call
           f(args...);
       }
   };
   
   // wrapper function
   template < typename... ArgsF, typename... ArgsT >
   void call_with_tuple(const std::function<void( ArgsF...)>& f,
           std::tuple<ArgsT...> const& t) {
       __apply_impl<sizeof...(ArgsT)>::apply(f, t);
   }
   

3. Message Dispatcher or Message Handler

   // message dispatcher
   class message_dispatcher {
   protected:
       // callback interface
       struct callback_t {
       public:
           virtual void call(void*) = 0;
       };
       // and implementation
       template< typename... Ty >
       struct callback_impl : public callback_t {
           typedef std::function< void(Ty...) > function_t;
   
           callback_impl(const function_t& f) {
               m_f = f;
           }
           virtual void call(void* data) {
               // parse to tuple
               auto t = parser < Ty... >::parse(data);
               // call function
               call_with_tuple(m_f, t);
           }
           function_t m_f;
       };
   public:
       // process incoming data 
       void process(int t, void* data) {
           m_c[t]->call(data);
       }
       // register callback for type t
       template< typename... Ty >
       void add(int t, const std::function< void(Ty...) >& f) {
           m_c[t] = new callback_impl < Ty... >(f);
       }
   protected:
       std::map< int, callback_t* > m_c;
   };
   

4. Example

   void foo(int a, float b, char c) {
       std::cout << "in foo(int,float,char) with args: ";
       std::cout << "1: " << a << ", "
               << "2: " << b << ", "
               << "3: " << c << std::endl;
   }
   
   struct foo_t {
       void foo(int a, double b) {
           std::cout << "in foo_t::foo(int,double) with args: ";
           std::cout << "1: " << a << ", "
                   << "2: " << b << std::endl;
       }
   };
   
   int main(int argc, char** argv) {
       // pack data 
       char* b1 = new char[size_of< int, float, char >::size];
       int a1 = 1;
       float a2 = 2.;
       char a3 = 'a';
       memcpy(b1, &a1, sizeof (a1));
       memcpy(b1 + sizeof (a1), &a2, sizeof (a2));
       memcpy(b1 + sizeof (a1) + sizeof (a2), &a3, sizeof (a3));
   
       // pack data 
       char* b2 = new char[size_of< int, double >::size];
       int a4 = 10;
       double a5 = 20.;
       memcpy(b2, &a4, sizeof (a4));
       memcpy(b2 + sizeof (a4), &a5, sizeof (a5));
   
       // create callbacks
       std::function<void(int, float, char) > f1(&foo);
       foo_t foo;
       std::function<void(int, double) > f2 = std::bind(&foo_t::foo, &foo,
               std::placeholders::_1,
               std::placeholders::_2);
   
       message_dispatcher md;
       // register callbacks
       md.add(0, f1);
       md.add(1, f2);
       // call 
       md.process(0, b1);
       md.process(1, b2);
   
       return 0;
   }
   

5. Output

   i 1
   f 2
   c a
   in foo(int,float,char) with args: 1: 1, 2: 2, 3: a
   i 10
   d 20
   in foo_t::foo(int,double) with args: 1: 10, 2: 20
   

Of course it work only with POD types. I have not used uint8_t, uint16_t and uint32_t, but there will be no problem with them.