Getting the size in bytes of nested stl containers

Question

I already know that for std::vector<int> I can do this:

std::vector<int> v;
// populate v
std::size_t bytes = sizeof(std::vector<int>) + sizeof(int) * v.size();

However, what is the correct way to get the size in bytes of std::vector<std::set<std::array<int, 10>>>? Do I have to call sizeof recursively to all nested containers? Example:

std::vector<std::set<std::array<int, 10>>> v;
// populate v
std::size_t bytes = sizeof(v);

for(auto& s : v)
{
    bytes += sizeof(s);

    for(auto& a : s)
    {
        bytes += sizeof(a) + sizeof(int) * 10;
    }
}

Or do I just use the same approach for std::vector<int> in the top level container?

First try to do that for `std::set` and who said that `sizeof(std::vector) + sizeof(int) * v.size();` gives you proper result — Slava, Aug 23 '18 at 17:40
If you care about actual memory footprint, maybe `capacity` would be a better indicator. But other containers may not be able to report their actual memory usage. Perhaps providing a custom allocator that can report memory usage would be more reliable. — François Andrieux, Aug 23 '18 at 17:43
@Slava I saw it here https://stackoverflow.com/a/17254457/3762200. — user3762200, Aug 23 '18 at 17:53
@user3762200 your calculation for vector is wrong. It should use `capacity`. It also exposes the fragility of what you are trying to achieve. Unless the class/container explicitly exposes some `mem_size` function, you won't have a reliable (and portable) way to find this out and would have to resort by case-by-case basis (where each stdlib version, each compiler, each OS and each container is a separate case in the matrix). For STL containers the closest you might get is by writing custom allocator that'd do the bookkeeping. — Dan M., Aug 29 '18 at 12:58

score 4 · Accepted Answer · answered Aug 23 '18 at 18:42

None of the standard C++ containers exposes an indicator on how much memory they allocate. It may seem viable to compute the allocated memory of a std::vector<T> v as sizeof(v) + v.capacity() * sizeof(T) (as std::vector<T> has to overallocate to meet its performance constraint using v.size() instead of v.capacity() would be wrong). However, there is no guarantee that this gives a correct value. While I don’t know of any standard library implementation which does it I can imagine implementing vector<T> with a pointer to the first element which is preceded with a control block storing the end, capacity, and allocator.

Most of the containers use internal nodes, e.g., pointers to previous and next node in a std::list or pointers to child nodes in a tree used by the ordered associative containers. There is no good estimate on the size of these nodes as the may contain additional control information. In addition, the memory allocations may use some information with allocated blocks, e.g., the size of allocated block.

The only potential way to estimate the allocated size of any of these data structures is to use a custom allocator which tracks the amount of memory used. Using this approach, of course, means that any subobject in the container correctly uses allocators for their resptive allocated memory, too. Note that some standard containers (e.g., std::array<T, N> do not properly forward allocators to their nested elements. In these cases there isn’t a way to determine the used memory correctly if the elements use allocations.

score 1 · Answer 2 · answered Aug 23 '18 at 19:25

First of all, there is no "easy" way of computing the storage of everything from the STL.

Following your try, the best I could think of is writing your own sizeof function which you will implement for everything you like, e.g.:

template<typename T>
size_t metaSizeOf(const T& t)
{
    return sizeof(T);
}

and then specialize it at will, e.g for a vector you would need something like:

template<typename T, typename Alloc, template <typename, typename> class V>
size_t metaSizeOf(const V<T, Alloc>& v)
{
    size_t bytes = sizeof(V<T, Alloc>);
    for(const auto& t: v) bytes += metaSizeOf(t);
    return bytes;
}

And copy/paste the same code with all template interfaces you have, e.g. for sets:

template<typename T, typename Compare, typename Alloc, template <typename, typename, typename> class V>
size_t metaSizeOf(const V<T, Compare, Alloc>& v)
{
    size_t bytes = sizeof(V<T, Compare, Alloc>);
    for(const auto& t: v) bytes += metaSizeOf(t);
    return bytes;
}

And so on. You can also write your own implementation for specific cases. For example for arrays you can write a generic function:

template<typename T, std::size_t N, template <typename, std::size_t> class V>
size_t metaSizeOf(const V<T, N>& v)
{
    size_t bytes = sizeof(V<T, N>);
    for(const auto& t: v) bytes += metaSizeOf(t);
    return bytes;
}

But you can also opt for a dedicated, optimized version for arrays only:

template<typename T, std::size_t N>
size_t metaSizeOf(const std::array<T, N>&)
{
    return sizeof(std::array<T, N>) + N * sizeof(T);
}

Example @ IdeOne.com

But as others pointed out, you will only count the size of the raw data, which is far from accurate. The memory usage of STL containers can be surprising.

Getting the size in bytes of nested stl containers

2 Answers2