Keeping fixed size symbols in shared memory

Question

I have been working on something which requires me store the list of symbols and the corresponding count, these are being stored as a string,int mapping. The problem is the input size is around 32 MB however when I try to store it in memory the size bloats to 1.4Gb, I went through a few links and found that std::string takes up a fair amount of memory for book keeping and is likely to cause the bloat, so I looked a little further and found that one could use the boost::array with a size equal to the max size of the symbol and since the boost array doesn't do any book keeping it doesn't cause memory bloat. However, I wish to place this boost array in the shared memory like so map<boost::array<char, 15>, int>.I took a look at the boost::array documentation and found that it doesn't support allocators, currently I am using allocators and segment managers from boost as follows:

typedef bip::allocator<char, bip::managed_shared_memory::segment_manager> CharAllocator;
typedef bip::basic_string<char, std::char_traits<char>, CharAllocator> SharedString;

I also came across this link, which seems to explain a lot: http://jovislab.com/blog/?p=89

Is there a way to do this, what are the other choices that I have. I am not looking for complete solutions, but hints. Thanks, Deb!

`std::string` does _not_ inherently bloat 32MB of data to 1.4GB!!! You did something hugely wrong in your code. — Lightness Races in Orbit, Oct 24 '14 at 17:47
_"I am not looking for complete solutions, but hints."_ This suggests that you do not mean to submit an entry to this repository of Q&As, but instead are looking for a discussion board, forum or chat room really. — Lightness Races in Orbit, Oct 24 '14 at 17:48
@surt I have not tested with std:allocator yet, please let me try that and come back with the results — deb, Oct 24 '14 at 17:59

score 1 · Accepted Answer · edited May 23 '17 at 12:31

Proper hints are:

use contiguous allocations
reserve up front

This answer contains a good side-by-side comparison of different allocation techniques with Boost managed_shared_memory:

How do I measure the size of a boost interprocess vector in shared memory?

In your particular case you could flatten the allocation by using

flat_map and
use pool allocator for the strings (or perhaps use boost::string_ref to refer to strings allocated from on large std::vector<char>)

Keeping fixed size symbols in shared memory

1 Answers1

Linked