How can I use C++11 variadic templates to define a vector-of-tuples backed by a tuple-of-vectors?

Question

Suppose I have a bunch of vectors:

vector<int> v1;
vector<double> v2;
vector<int> v3;

all of the same length. Now, for every index i, I would like to be able to treat (v1[i], v2[i], v3[i]) as a tuple, and maybe pass it around. In fact, I want to have a a vector-of-tuples rather than a tuple-of-vectors, using which I can do the above. (In C terms, I might say an array-of-structs rather than a struct-of-arrays). I do not want to effect any data reordering (think: really long vectors), i.e. the new vector is backed by the individual vectors I pass in. Let's .

Now, I want the class I write (call it ToVBackedVoT for lack of a better name) to support any arbitrary choice of vectors to back it (not just 3, not int, double and int, not every just scalars). I want the vector-of-tuples to be mutable, and for no copies to be made on construction/assignments.

If I understand correctly, variadic templates and the new std::tuple type in C++11 are the means for doing this (assuming I don't want untyped void* arrays and such). However, I only barely know them and have never worked with them. Can you help me sketch out how such a class will look like? Or how, given

template <typename ... Ts>

I can express something like "the list of template arguments being the replacement of each typename in the original template arguments with a vector of elements of this type"?

Note: I think I might also want to later be able to adjoin additional vectors to the backing vectors, making an instance of ToVBackedVoT<int, double, int> into, say, an instance of ToVBackedVoT<int, double, int, unsigned int>. So, bear that in mind when answering. This is not critically important though.

Answer 1

One idea is to keep the storage in the "struct of array" style in form of vectors for good performance if only a subset of the fields are used for a particular task. Then, for each kind of task requiring a different set of fields, you can write a lightweight wrapper around some of those vectors, giving you a nice random access iterator interface similar to what std::vector supports.

Concerning the syntax of variadic templates, this is how a wrapper class (without any iterators yet) could look like:

template<class ...Ts> // Element types
class WrapMultiVector
{
    // references to vectors in a TUPLE
    std::tuple<std::vector<Ts>&...> m_vectors;

public:
    // references to vectors in multiple arguments
    WrapMultiVector(std::vector<Ts> & ...vectors)
        : m_vectors(vectors...)    // construct tuple from multiple args.
    {}
};

To construct such a templated class, it's often preferred to have a template type deducting helper function available (similar to those make_{pair|tuple|...} functions in std):

template<class ...Ts> // Element types
WrapMultiVector<Ts...> makeWrapper(std::vector<Ts> & ...vectors) {
    return WrapMultiVector<Ts...>(vectors...);
}

You already see different types of "unpacking" the type list.

Adding iterators suitable to your application (you requested in particular random access iterators) is not so easy. A start could be forward only iterators, which you might extend to random access iterators.

The following iterator class is capable of being constructed using a tuple of element iterators, being incremented and being dereferenced to obtain a tuple of element references (important for read-write access).

class iterator {
    std::tuple<typename std::vector<Ts>::iterator...> m_elemIterators;

public:
    iterator(std::tuple<typename std::vector<Ts>::iterator...> elemIterators) 
        : m_elemIterators(elemIterators)
    {}

    bool operator==(const iterator &o) const {
        return std::get<0>(m_elemIterators) == std::get<0>(o.m_elemIterators);
    }
    bool operator!=(const iterator &o) const {
        return std::get<0>(m_elemIterators) != std::get<0>(o.m_elemIterators);
    }

    iterator& operator ++() {
        tupleIncrement(m_elemIterators);
        return *this;
    }
    iterator operator ++(int) {
        iterator old = *this;
        tupleIncrement(m_elemIterators);
        return old;
    }

    std::tuple<Ts&...> operator*() {
        return getElements(IndexList());
    }

private:
    template<size_t ...Is>
    std::tuple<Ts&...> getElements(index_list<Is...>) {
        return std::tie(*std::get<Is>(m_elemIterators)...);
    }
};

For demonstration purposes, two different patterns are in this code which "iterate" over a tuple in order to apply some operation or construct a new tuple with some epxression to be called per element. I used both in order to demonstrate alternatives; you can also use the second method only.

1. tupleIncrement: You can use a helper function which uses meta programming to index a single entry and advance the index by one, then calling a recursive function, until the index is at the end of the tuple (then there is a special case implementation which is triggered using SFINAE). The function is defined outside of the class and not above; here is its code:

   template<std::size_t I = 0, typename ...Ts>
   inline typename std::enable_if<I == sizeof...(Ts), void>::type
   tupleIncrement(std::tuple<Ts...> &tup)
   { }
   template<std::size_t I = 0, typename ...Ts>
   inline typename std::enable_if<I < sizeof...(Ts), void>::type
   tupleIncrement(std::tuple<Ts...> &tup)
   {
       ++std::get<I>(tup); 
       tupleIncrement<I + 1, Ts...>(tup);
   }
   

   This method can't be used to assign a tuple of references in the case of operator* because such a tuple has to be initialized with references immediately, which is not possible with this method. So we need something else for operator*:

2. getElements: This version uses an index list (https://stackoverflow.com/a/15036110/592323) which gets expanded too and then you can use std::get with the index list to expand full expressions. The IndexList when calling the function instantiates an appropriate index list which is only required for template type deduction in order to get those Is.... The type can be defined in the wrapper class:

   // list of indices
   typedef decltype(index_range<0, sizeof...(Ts)>()) IndexList;
   

More complete code with a little example can be found here: http://ideone.com/O3CPTq

Open problems are:

• If the vectors have different sizes, the code fails. Better would be to check all "end" iterators for equality; if one iterator is "at end", we're also "at end"; but this would require some logic more than operator== and operator!= unless it's ok to "fake" it in; meaning that operator!= could return false as soon as any operator is unequal.

• The solution is not const-correct, e.g. there is no const_iterator.

• Appending, inserting etc. is not possible. The wrapper class could add some insert or and / or push_back function in order to make it work similar to std::vector. If your goal is that it's syntactically compatible to a vector of tuples, reimplement all those relevant functions from std::vector.

• Not enough tests ;)

Answer 2

An alternative to all the variadic template juggling is to use the boost::zip_iterator for this purpose. For example (untested):

std::vector<int> ia;
std::vector<double> d;
std::vector<int> ib;

std::for_each(
  boost::make_zip_iterator(
    boost::make_tuple(ia.begin(), d.begin(), ib.begin())
    ),
  boost::make_zip_iterator(
    boost::make_tuple(ia.end(), d.end(), ib.end())
    ),
  handle_each()
  );

Where your handler, looks like:

struct handle_each :
  public std::unary_function<const boost::tuple<const int&, const double&, const int&>&, void>
{
  void operator()(const boost::tuple<const int&, const double&, const int&>& t) const
  {
    // Now you have a tuple of the three values across the vector...
  }
};

As you can see, it's pretty trivial to expand this to support an arbitrary set of vectors..

Answer 3

You may use something like:

#if 1 // Not available in C++11, so write our own

// class used to be able to use std::get<Is>(tuple)...
template<int... Is>
struct index_sequence { };

// generator of index_sequence<Is>
template<int N, int... Is>
struct make_index_sequence : make_index_sequence<N - 1, N - 1, Is...> { };

template<int... Is>
struct make_index_sequence<0, Is...> : index_sequence<Is...> { };

#endif

// The 'converting' class
// Note that it doesn't check that vector size are equal...
template<typename ...Ts>
class ToVBackedVoT
{
public:
    explicit ToVBackedVoT(std::vector<Ts>&... vectors) : data(vectors...) {}

    std::tuple<const Ts&...> operator [] (unsigned int index) const
    {
        return at(index, make_index_sequence<sizeof...(Ts)>());
    }
    std::tuple<Ts&...> operator [] (unsigned int index)
    {
        return at(index, make_index_sequence<sizeof...(Ts)>());
    }

private:
    template <int... Is>
    std::tuple<const Ts&...> at(unsigned int index, index_sequence<Is...>) const
    {
        return std::tie(std::get<Is>(data)[index]...);
    }

    template <int... Is>
    std::tuple<Ts&...> at(unsigned int index, index_sequence<Is...>)
    {
        return std::tie(std::get<Is>(data)[index]...);
    }

private:
    std::tuple<std::vector<Ts>&...> data;
};

And to iterate, create an 'IndexIterator' like the one in https://stackoverflow.com/a/20272955/2684539

To adjoin additional vectors, you have to create an other ToVBackedVoT as std::tuple_cat does for std::tuple

Answer 4

From asker's clarification on how this would be used (code that takes a tuple), I'm going to propose this instead.

//give the i'th element of each vector
template<typename... Ts>
inline tuple<Ts&...> ith(size_t i, vector<Ts>&... vs){
    return std::tie(vs[i]...);
}

---

There's a proposal to allow parameter packs to be saved as members of classes (N3728). Using that, here's some untested and untestable code.

template<typename... Types>
class View{
private:
    vector<Types>&... inner;

public:

    typedef tuple<Types&...> reference;

    View(vector<Types>&... t): inner(t...) {}

    //return smallest size
    size_t size() const{
        //not sure if ... works with initializer lists
        return min({inner.size()...});
    }

    reference operator[](size_t i){
        return std::tie(inner[i]...);
    }
};

And iteration:

public:
    iterator begin(){
        return iterator(inner.begin()...);
    }
    iterator end(){
        return iterator(inner.end()...);
    }

    //for .begin() and .end(), so that ranged-based for can be used
    class iterator{
        vector<Types>::iterator... ps;
        iterator(vector<Types>::iterator... its):ps(its){}
        friend View;

    public:

        //pre:
        iterator operator++(){
            //not sure if this is allowed.
            ++ps...;
            //use this if not:
            //  template<typename...Types> void dummy(Types... args){} //global
            //  dummy(++ps...);
            return *this;
        }
        iterator& operator--();
        //post:
        iterator operator++(int);
        iterator operator--(int);
        //dereference:
        reference operator*()const{
            return std::tie(*ps...);
        }
        //random access:
        iterator operator+(size_t i) const;
        iterator operator-(size_t i) const;
        //need to be able to check end
        bool operator==(iterator other) const{
            return std::make_tuple(ps...) == std::make_tuple(other.ps...);
        }
        bool operator!=(iterator other) const{
            return std::make_tuple(ps...) != std::make_tuple(other.ps...);
        }

    };

Answer 5

Conversion to a std::tuple of vectors (vector::iterators):

#include <iostream>
#include <vector>

// identity
// ========

struct identity
{
    template <typename T>
    struct apply {
        typedef T type;
    };
};

// concat_operation
// ================

template <typename Operator, typename ...> struct concat_operation;

template <
    typename Operator,
    typename ...Types,
    typename T>
struct concat_operation<Operator, std::tuple<Types...>, T>
{
    private:
    typedef typename Operator::template apply<T>::type concat_type;
    public:
    typedef std::tuple<Types..., concat_type> type;
};

template <
    typename Operator,
    typename ...Types,
    typename T,
    typename ...U>
struct concat_operation<Operator, std::tuple<Types...>, T, U...>
{
    private:
    typedef typename Operator::template apply<T>::type concat_type;
    public:
    typedef typename concat_operation<
        Operator,
        std::tuple<Types..., concat_type>,
        U...>
    ::type type;
};

template <
    typename Operator,
    typename T,
    typename ...U>
struct concat_operation<Operator, T, U...>
{
    private:
    typedef typename Operator::template apply<T>::type concat_type;
    public:
    typedef typename concat_operation<
        Operator,
        std::tuple<concat_type>,
        U...>
    ::type type;
};


// ToVectors (ToVBackedVoT)
// =========

template <typename ...T>
struct ToVectors
{
    private:
    struct to_vector {
        template <typename V>
        struct apply {
            typedef typename std::vector<V> type;
        };
    };

    public:
    typedef typename concat_operation<to_vector, T...>::type type;
};

// ToIterators
// ===========

template <typename ...T>
struct ToIterators;

template <typename ...T>
struct ToIterators<std::tuple<T...>>
{
    private:
    struct to_iterator {
        template <typename V>
        struct apply {
            typedef typename V::iterator type;
        };
    };

    public:
    typedef typename concat_operation<to_iterator, T...>::type type;
};


int main() {
    typedef ToVectors<int, double, float>::type Vectors;
    typedef ToVectors<Vectors, int, char, bool>::type MoreVectors;
    typedef ToIterators<Vectors>::type Iterators;

    // LOG_TYPE(Vectors);
    // std::tuple<
    //     std::vector<int, std::allocator<int> >,
    //     std::vector<double, std::allocator<double> >,
    //     std::vector<float, std::allocator<float> > >

    // LOG_TYPE(Iterators);
    // std::tuple<
    //     __gnu_cxx::__normal_iterator<int*, std::vector<int, std::allocator<int> > >,
    //     __gnu_cxx::__normal_iterator<double*, std::vector<double, std::allocator<double> > >,
    //     __gnu_cxx::__normal_iterator<float*, std::vector<float, std::allocator<float> > > >
}

Answer 6

As an alternative similar to boost::zip_iterator I wrote a zip function with a very simple interface:

vector<int> v1;
vector<double> v2;
vector<int> v3;

auto vec_of_tuples = zip(v1, v2, v3);

For example, iterate over these tuples:

for (auto tuple : zip(v1, v2, v3)) {
    int x1; double x2; int x3;
    std::tie(x1, x2, x3) = tuple;
    //...
}

Here, zip() takes any number of ranges of any type. It returns an adaptor which can be seen as a lazily evaluated range over a tuple of elements originating from the wrapped ranges.

The adaptor is part of my Haskell-style functional library "fn" and implemented using variadic templates.

Currently it doesn't support modification of the original ranges' values via the adaptor because of the design of the library (it's intended to be used with non-mutable ranges like in functional programming).

A brief explanation on how this is done is: zip(...) returns an adaptor object which implements begin() and end(), returning an iterator object. The iterator holds a tuple of iterators to the wrapped ranges. Incrementing the iterator increments all wrapped iterators (which is implemented using an index list and unpacking an incrementing expression into a series of expressions: ++std::get<I>(iterators)...). Dereferencing the iterator will decrement all wrapped iterators and pass it to std::make_tuple (which is also implemented as unpacking the expression *std::get<I>(iterators)...).

P.S. Its implementation is based on a lot of ideas coming from answers to this question.