Reshape vector data stream to matrix for large amounts of data, equivalent to np.reshape

Question

I have data from a sensor I need to put into a 3D matrix, the data comes in a 1d buffer of uint16.

I am provided with meta data informing me of the dimensions P, R, D such that the 1d buffer is of length PRD and the resulting 3D matrix should be of size PxRxD. The length of the buffer will typically be on the order of 1e6 points.

I'd like to access, store, and manipulate this using C++. But I'm not entirely sure how best to go about doing so. I feel there must be a library for this already, so I'd like to avoid having to implement some C-array or std::vector custom implementation. Perhaps Eigen could be possible, (given they have reshaping from Vector to Matrix implemented) but I'm not sure how well it will handle these sizes.

I have made a python script to demo out this functionality, and in python the data comes in as tuples, so an approach which works is np.asarray(tuples, dtype=np.uint16).reshape(P, R, D). I'll just eventually need to move away from python for speed, so a 100% C++ implementation is desirable.

Edit: Some extra details I gathered from responses could be useful

data is coming out of a sensor, so will be read from ethernet connection. So some memory for this container will need to be allocated locally. At least the 1d buffer I think can't be re-used
Ill need to do signal processing on the data, so formats which enable that are preferred, mostly it'll be processing RxD matrices for each P.
P is typically ~12, R and D will each be on the order of 1e2.

There are certainly going to be libraries that do this, but at the core is `row_index * number_of_columns + column_index`. All you need is a function that performs the mapping for you or to wrap the 1D array in a class that conceals the array's true nature and presents a 2D interface. Note that at the end of the day, all arrays in C++ are 1D. Just sometimes they are a 1D array of 1D arrays. — user4581301, Jun 14 '23 at 18:21
[Here is a simple example of a wrapper](https://stackoverflow.com/a/36123944/4581301). Replace `vector v;` with a pointer to the 1D array and the rest comes out in the wash. — user4581301, Jun 14 '23 at 18:23

Homer512 · Answer 1 · 2023-06-15T17:35:54.013

Well, an Eigen implementation is a bit awkward because the library is limited to 2D. There is an unsupported tensor module for higher dimensional data. If you want to stick with the main library, something like this works:

#include <Eigen/Dense>

#include <cassert>
#include <cstdint>
#include <vector>


class Tensor3d
{
public:
    using Scalar = std::uint16_t;
    using Vector = std::vector<Scalar>;
    using Matrix = Eigen::Matrix<Scalar, Eigen::Dynamic, Eigen::Dynamic>;
    using MatrixView = Eigen::Map<Matrix>;
    using ConstMatrixView = Eigen::Map<const Matrix>;
private:
    VectorView buffer;
    Eigen::Index rows, cols, depth;
public:
    /** Takes ownership of an std::vector<uint16_t> */
    Tensor3d(Vector&& buffer,
        Eigen::Index rows /*=D*/, Eigen::Index cols /*=R*/,
        Eigen::Index depth /*=P*/)
    : buffer(std::move(buffer)),
      rows(rows), cols(cols), depth(depth)
    {
        assert(buffer.size() == static_cast<std::size_t>(
               rows * cols *depth);
    }
    /** Gets one plane as a 2D matrix view */
    MatrixView operator[](Eigen::Index plane) noexcept
    {
        assert(plane >= 0 && plane < depth);
        Eigen::Index offset = plane * rows * cols;
        return MatrixView(buffer.data() + offset, rows, cols);
    }
    ConstMatrixView operator[](Eigen::Index plane) const noexcept
    {
        assert(plane >= 0 && plane < depth);
        Eigen::Index offset = plane * rows * cols;
        return ConstMatrixView(buffer.data() + offset, rows, cols);
    }
};

void foo(Tensor3d& tensor)
{
    tensor[3] *= 3;
    Eigen::Index p = 1, r = 2, d = 3;
    std::uint16_t value = tensor[p](d, r);
    auto all_d_of_p_and_r = tensor[p].col(r);
    auto all_r_of_p_and_d = tensor[p].row(d);
}

Note that Eigen matrices are column-major (in Python this would be known as Fortran order), so rows are the innermost dimension.

I'm not sure why but I thought Eigen had 3d possibilities as well, I've added some details in an edit for what you said was unclear. Maybe a vector of matrices could be possible, given P << R and D — Morten Nissov, Jun 14 '23 at 20:45
@MortenNissov vector of matrices isn't a good approach. It's like list of arrays in Python. You don't normally do that — Homer512, Jun 14 '23 at 21:31
Given I know P to be small (~12) and most operations of interest are on the RxD matrix, it still doesn't make sense? — Morten Nissov, Jun 15 '23 at 04:49
@MortenNissov I'm still a bit unclear about your design. If you need to copy the data anyway copying into a `vector>` is fine. Slightly inefficient but not too bad. But "borrowing" the original buffer into a `vector>` doesn't work well with Eigen's design of the `Map` type — Homer512, Jun 15 '23 at 06:49
From the sensor SDK I will get a std::vector of length P*R*D (P~12). Sorry I'm not sure entirely what information I should be providing to clear up the confusion. — Morten Nissov, Jun 15 '23 at 15:57
@MortenNissov I've adjusted the code to how I would deal with it. It sounds like you just need a wrapper that takes the vector and then can give you matrix-like views of the P slices. These are all cheap operations that can happen on-the-fly — Homer512, Jun 15 '23 at 17:37

Reshape vector data stream to matrix for large amounts of data, equivalent to np.reshape

1 Answers1