Is it possible to initialize std::vector over already allocated memory?

Question

My question is fairly simple and I am quite surprised I can't find anything related. Probably it is easy or totally stupid (or I can't search).

As the title says, is it possible to use std::vector on already allocated memory, so it does not allocate new elements from start but uses what is given. I would imagine it as something like:

T1 *buffer = new T1[some_size];
std::vector<T2> v(buffer, some_size); // <- ofc doesn't work

The opposite is quite simple and (maybe not pretty but) works:

std::vector<T2> v(some_size);
T1 *buffer = &v[0];

The storage is guaranteed to be continuous, so it is as safe as an iterator.

My motivation is quite simple. I pass around some raw memory data, i.e. bytes, and since I know their interpretation at some other places I would like to convert them back to something meaningful. I could do a reinterpret_cast and use normal c-style array, but I prefer c++ facilities.

I get the feeling this should be safe given that we give up ownership of buffer to vector, because it needs to be able to reallocate.

Answer 1

Like this.. Containers in the standard usually take an allocator. Using c++11's allocator traits, it is very easy to create an allocator as you don't have to have all the members in the allocator. However if using an older version of C++, you will need to implement each member and do the rebinding as well!

For Pre-C++11, you can use the following:

#include <iterator>
#include <vector>
#include <iostream>

template<typename T>
class PreAllocator
{
    private:
        T* memory_ptr;
        std::size_t memory_size;

    public:
        typedef std::size_t size_type;
        typedef ptrdiff_t difference_type;
        typedef T* pointer;
        typedef const T* const_pointer;
        typedef T& reference;
        typedef const T& const_reference;
        typedef T value_type;


        PreAllocator(T* memory_ptr, std::size_t memory_size) throw() : memory_ptr(memory_ptr), memory_size(memory_size) {};
        PreAllocator (const PreAllocator& other) throw() : memory_ptr(other.memory_ptr), memory_size(other.memory_size) {};

        template<typename U>
        PreAllocator (const PreAllocator<U>& other) throw() : memory_ptr(other.memory_ptr), memory_size(other.memory_size) {};

        template<typename U>
        PreAllocator& operator = (const PreAllocator<U>& other) {return *this;}
        PreAllocator<T>& operator = (const PreAllocator& other) {return *this;}
        ~PreAllocator() {}

        pointer address (reference value) const {return &value;}
        const_pointer address (const_reference value) const {return &value;}

        pointer allocate (size_type n, const void* hint = 0) {return memory_ptr;}
        void deallocate (T* ptr, size_type n) {}

        void construct (pointer ptr, const T& val) {new (ptr) T (val);}

        template<typename U>
        void destroy (U* ptr) {ptr->~U();}
        void destroy (pointer ptr) {ptr->~T();}

        size_type max_size() const {return memory_size;}

        template<typename U>
        struct rebind
        {
            typedef PreAllocator<U> other;
        };
};

int main()
{
    int my_arr[100] = {0};
    std::vector<int, PreAllocator<int> > my_vec(PreAllocator<int>(&my_arr[0], 100));
    my_vec.push_back(1024);
    std::cout<<"My_Vec[0]: "<<my_vec[0]<<"\n";
    std::cout<<"My_Arr[0]: "<<my_arr[0]<<"\n";

    int* my_heap_ptr = new int[100]();
    std::vector<int, PreAllocator<int> > my_heap_vec(PreAllocator<int>(&my_heap_ptr[0], 100));
    my_heap_vec.push_back(1024);
    std::cout<<"My_Heap_Vec[0]: "<<my_heap_vec[0]<<"\n";
    std::cout<<"My_Heap_Ptr[0]: "<<my_heap_ptr[0]<<"\n";

    delete[] my_heap_ptr;
    my_heap_ptr = NULL;
}

For C++11, you can use the following:

#include <cstdint>
#include <iterator>
#include <vector>
#include <iostream>

template <typename T>
class PreAllocator
{
    private:
        T* memory_ptr;
        std::size_t memory_size;

    public:
        typedef std::size_t     size_type;
        typedef T*              pointer;
        typedef T               value_type;

        PreAllocator(T* memory_ptr, std::size_t memory_size) : memory_ptr(memory_ptr), memory_size(memory_size) {}

        PreAllocator(const PreAllocator& other) throw() : memory_ptr(other.memory_ptr), memory_size(other.memory_size) {};

        template<typename U>
        PreAllocator(const PreAllocator<U>& other) throw() : memory_ptr(other.memory_ptr), memory_size(other.memory_size) {};

        template<typename U>
        PreAllocator& operator = (const PreAllocator<U>& other) { return *this; }
        PreAllocator<T>& operator = (const PreAllocator& other) { return *this; }
        ~PreAllocator() {}


        pointer allocate(size_type n, const void* hint = 0) {return memory_ptr;}
        void deallocate(T* ptr, size_type n) {}

        size_type max_size() const {return memory_size;}
};

int main()
{
    int my_arr[100] = {0};
    std::vector<int, PreAllocator<int>> my_vec(0, PreAllocator<int>(&my_arr[0], 100));
    my_vec.push_back(1024);
    std::cout<<"My_Vec[0]: "<<my_vec[0]<<"\n";
    std::cout<<"My_Arr[0]: "<<my_arr[0]<<"\n";

    int* my_heap_ptr = new int[100]();
    std::vector<int, PreAllocator<int>> my_heap_vec(0, PreAllocator<int>(&my_heap_ptr[0], 100));
    my_heap_vec.push_back(1024);
    std::cout<<"My_Heap_Vec[0]: "<<my_heap_vec[0]<<"\n";
    std::cout<<"My_Heap_Ptr[0]: "<<my_heap_ptr[0]<<"\n";

    delete[] my_heap_ptr;
    my_heap_ptr = nullptr;
}

Notice the difference between the two allocators! This will work with both heap buffers/arrays and stack buffer/arrays. It will also work with most containers. It is safer to use the Pre-C++11 version because it will be backwards compatible and work with more containers (ie: std::List).

You can just place the allocator in a header and use it as much as you want in any projects. It is good if you want to use SharedMemory or any buffer that is already allocated.

WARNING: DO NOT use the same buffer for multiple containers at the same time! A buffer can be reused but just make sure no two containers use it at the same time.

Example:

int my_arr[100] = {0};
std::vector<int, PreAllocator<int> > my_vec(PreAllocator<int>(&my_arr[0], 100));
std::vector<int, PreAllocator<int> > my_vec2(PreAllocator<int>(&my_arr[0], 100));

my_vec.push_back(1024);
my_vec2.push_back(2048);

std::cout<<"My_Vec[0]: "<<my_vec[0]<<"\n";
std::cout<<"My_Arr[0]: "<<my_arr[0]<<"\n";

The output of the above is 2048! Why? Because the last vector overwrote the values of the first vector since they share the same buffer.

Answer 2

Yes, std::vector takes a custom allocator as a template parameter which can achieve what you want.

Answer 3

If you look at the docs for std::vector you will see the second template parameter is a custom allocator:

template < class T, class Alloc = allocator<T> > class vector;

You can define your own std::allocator that returns whatever memory you want. This may be a lot more work than is worth it for your purposes though.

You cannot just pass in a pointer to random memory, however. You would have problems like, what if the vector needs to grow beyond the size of your initial buffer?

If you just want to operate on raw bytes sometimes, and vectors at other times, I would write helper functions to convert between the two, and just convert between the two when needed. If this causes performance issues (which you should measure), then custom allocator is your next course of action.