4

I'm right now getting into shared memory using boost::interprocess.

I've defined a few std::unordered_map and std::unordered_set types like in the following manner:

#include <boost/interprocess/allocators/allocator.hpp>

#include <unordered_map> // NOT the boost implementation ...

...

namespace ipc = boost::interprocess;

 /**
 * allocator type needed to construct maps in shared memory
 */
typedef ipc::allocator<std::pair<const size_t, std::string>,
           ipc::managed_shared_memory::segment_manager> OBJ_MAP_ALLOCATOR;

/**
 * map type to construct maps in shared memory
 */
typedef std::unordered_map<size_t,
               std::string,
               std::hash<size_t>,
               std::equal_to<size_t>,
               OBJ_MAP_ALLOCATOR> OBJ_MAP_TYPE;

I initialised them like this:

ipc::managed_shared_memory segment;

// allocate segment etc ... 

OBJ_MAP_ALLOCATOR alloc_inst(segment.get_segment_manager());
objMap = segment.find_or_construct<OBJ_MAP_TYPE> (ipc::unique_instance)(alloc_inst);

This seems to work fine, i haven't found any problems during compile- or runtime (working on macOS, Apple LLVM version 9.1.0 (clang-902.0.39.1), with C++14 standard).

In the Boost documentation, only the Boost containers, or the interprocess-specific implementations are mentioned. Unfortunately, they do not seem to contain the unordered versions.

So, i wonder if there's anything problematic about using the default STL containers with the Boost allocators ? Maybe on a different platform ?

Any hint appreciated !

Update:

I was wondering if it was working in a different environment, so i wrote a minimal example on Coliru (which surprisingly works with std::string):

http://coliru.stacked-crooked.com/a/91d1a143778cf3e9

EllipsenPark
  • 127
  • 9
  • I am not an expert with boost allocators, but the whole idea of separating allocators and containers is about the fact that those two should be completely independent of each other. Because of that, I would be extremely surprised if boost allocators work with some containers, but not the others. – SergeyA Apr 09 '18 at 15:48
  • 1
    You can't use std::string with shared memory. You need to instantiate your own basic_string with the right allocator. – rustyx Apr 09 '18 at 16:04
  • @SergeyA It's not so much "the boost allocators" that are a challenge to many container libraries, but "stateful allocators" which these allocators happen to be an example of :) [See my answer] – sehe Apr 09 '18 at 18:16

1 Answers1

9

unordered_map will cope with Boost Interprocess allocators IFF your library implementation has support for stateful allocators¹ and allocators using non-raw pointer types.

Even so, like @rustyx mentions, you're going to be in deep trouble if you actually share the memory with another process. The other process is likely to map the segment at a different base address, making all pointers stored inside the memory region invalid.

☞ You need to use a Interprocess allocator with the string too!

Here's what I usually prefer to write:

#include <boost/interprocess/allocators/allocator.hpp>
#include <boost/interprocess/containers/string.hpp>
#include <boost/interprocess/managed_shared_memory.hpp>

#include <unordered_map>

namespace ipc = boost::interprocess;

namespace Shared {
    using Segment = ipc::managed_shared_memory;
    using Manager = Segment::segment_manager;
    template <typename T> using Alloc = ipc::allocator<T, Manager>;
    template <typename K, typename V, typename KH = std::hash<K>, typename KEq = std::equal_to<K> >
        using HashMap = std::unordered_map<K, V, KH, KEq, Alloc<std::pair<const K, V>> >;

    using String = ipc::basic_string<char, std::char_traits<char>, Alloc<char> >;
}

using OBJ_MAP_TYPE = Shared::HashMap<size_t, Shared::String>;

int main() {
    Shared::Segment msm(ipc::open_or_create, "test", 10ul<<20);

    Shared::Manager* mgr = msm.get_segment_manager();
    OBJ_MAP_TYPE& m = *msm.find_or_construct<OBJ_MAP_TYPE>("aname")(msm.get_segment_manager());

    m.emplace(42, Shared::String("LtUaE", msm.get_segment_manager()));
}

Notable details:

  1. This bit:

    Shared::Manager* mgr = msm.get_segment_manager();
    OBJ_MAP_TYPE& m = *msm.find_or_construct<OBJ_MAP_TYPE>("aname")(mgr);
    

    is a convenient short-cut for doing:

    Shared::Alloc<OBJ_MAP_TYPE::value_type> alloc_inst(msm.get_segment_manager());
    OBJ_MAP_TYPE& m = *msm.find_or_construct<OBJ_MAP_TYPE>("aname")(alloc_inst);
    

    This works because the implicit conversion from segment-manager pointer to allocator instance is allowed.

Enter MAGIC

You'll note that the nested allocator is clumsy to work with:

m.emplace(42, Shared::String("LtUaE", msm.get_segment_manager()));

That's what the designers of scoped_allocator_adaptor tried to solve. If you change the allocator into:

template <typename T> using Alloc = std::scoped_allocator_adaptor<ipc::allocator<T, Manager> >;

You can suddenly just write:

m.emplace(42, "LtUaE");

This is because in-place construction is defined in terms of uses- allocator construction (see [allocator.uses.construction])

See it Live On Coliru


¹ prepare to be surprised, @SergeyA. Libstdc++ didn't support this last time I checked, but its unordered_map supports it since GCC 4.9.0, and OP seems to have anecdotal evidence that libc++ does (although we don't even know whether there was ever an instance of the typedef :))

Jonathan Wakely
  • 166,810
  • 27
  • 341
  • 521
sehe
  • 374,641
  • 47
  • 450
  • 633
  • 1
    I forgot to mention the drawbacks... There are a few: [see this conversation from 2017 in `Lounge`](https://chat.stackoverflow.com/rooms/10/conversation/boost-interprocess-segment-woes-bip). So, portability, versioning, growing/resizing are pretty hairy issues with whatever boost delivers out of the box. YMMV! – sehe Apr 09 '18 at 18:15
  • Such a great answer and no upvotes... It's a shame. Doing my part. – SergeyA Apr 09 '18 at 19:00
  • @SergeyA Cheers! Life in the [tag:boost] tag is a patience game (as in all low-traffic tags, I suppose) – sehe Apr 09 '18 at 19:12
  • 1
    @sehe Thanks for the exhaustive answer :) To elaborate on the anecdotal evidence: 1.) Yes, i used multiple instances of the typedef, just the way i've written them with `std::string` 2.) i'm not in deep trouble ... at least in my environment it works just fine with `std::string` ... now i read that it shouldn't, which is what puzzles me in the first place ... – EllipsenPark Apr 10 '18 at 06:43
  • After gaining some more experience with this topic, it seems that using `std::string` limits the length the values you can store. I'm not 100% sure what the exact conditions are, but when using `std::string` i was limited to values less than 16 characters, and would get a segfault when trying to read them ... – EllipsenPark Jul 17 '18 at 09:03
  • @EllipsenPark No. That's accidental. If your `std::string` implementation happens to have SSO you'll find that it *hides* your error of not using the correct allocator. – sehe Jul 17 '18 at 09:07
  • @EllipsenPark see here e.g. https://stackoverflow.com/questions/33513051/structures-and-vectors-in-boost-shared-memory/33518518#33518518 and https://stackoverflow.com/questions/48675001/boost-interprocess-cout-a-string-variable-when-iterating-through-a-map-that-ref/48696596#48696596 (and more like https://stackoverflow.com/questions/36831496/unordered-map-with-string-in-managed-shared-memory-fails/36835162#36835162) – sehe Jul 17 '18 at 09:12
  • Worth nothing that `Alloc` is a GNU extension (e.g. `-std=gnu++17`) and won't work with `-std=c++17`. https://gcc.gnu.org/bugzilla/show_bug.cgi?id=86977 – João Neto Aug 16 '18 at 10:15
  • 1
    _"specifies `Alloc` because the actual node-type allocated by the map is implementation-defined"_ Wat? You don't need to know the unspecified internal node type, you pass an allocator with the same `value_type` as the container's `value_type`, as for every use of allocators with standard containers. So `Alloc>`. See the first row of the table of [allocator-aware container requirements](https://en.cppreference.com/w/cpp/named_req/AllocatorAwareContainer). – Jonathan Wakely Aug 16 '18 at 10:31
  • @JonathanWakely sure. That's equivalent. And because `void` used to work it was simply a shorter and less potentially-misleading spelling for the same. OT: I hate when people act all outraged/baffled/upset over chump change like this ("Wat?"). – sehe Aug 16 '18 at 10:35
  • 1
    @joaomlneto I'll try to remember to update the answer when I get back in the vicinity of keyboards – sehe Aug 16 '18 at 10:36
  • No it's not equivalent, because one is a non-standard extension not supported by all compilers or in all modes, and one is valid C++. The "wat?" is because of the entirely bogus justification based on the node type, which you don't need to know. – Jonathan Wakely Aug 16 '18 at 10:36
  • Sigh. You can edit the answer for me instead of this comment fighting. That way I can't forget. – sehe Aug 16 '18 at 10:38
  • 1
    Regarding the first footnote, libstdc++'s `unordered_map` got stateful allocator support in 2013, for the 4.9.0 release. – Jonathan Wakely Aug 16 '18 at 10:38
  • @JonathanWakely is that specific to `unordered_map`/`vector` or is it something general? https://stackoverflow.com/a/51895623/4288486 – João Neto Aug 18 '18 at 23:53
  • this code compiles fine on my x64 ubuntu machine, running exe for the first time seems OK, but a segfault occurs when i run it again. anybody know why ? – frank Jun 23 '20 at 17:47
  • @frank Huh. I can confirm: it's broken with libstdc++ `unordered_map`. Use `boost::unordered_map` for now. Still okay [5.5.0](https://wandbox.org/permlink/zjEl4GngtXAH93Ey), [6.3.0](https://wandbox.org/permlink/Xxlbx6TdmGkujNus) which uses boost 1.73 already, [7.3.0](https://wandbox.org/permlink/aAqV83F5Fe47Uj1h), [8.3.0](https://wandbox.org/permlink/kaO4p5y1YEg2UiD2), [9.3.0](https://wandbox.org/permlink/WaHnzXECOc91GHWD), [10.1.0](https://wandbox.org/permlink/Uc1Sco0DG0F8WMkM) even, [gcc HEAD](https://wandbox.org/permlink/DzzTpFSCTG6tkrPV). But on my Ubuntu 18.04 it's broken. – sehe Jun 23 '20 at 20:45
  • The Wandbox test might not be reliable, since on my system it can appear to work within a single run (maybe due to same ASLR I think), but ASAN is still triggered. I cannot trigger ASAN using clang + libc++, but clang does trigger ASAN using libstdc++, confirming a problem with libstd++. – sehe Jun 23 '20 at 21:01
  • Sadly Wandbox [doesn't seem to allow asan](https://wandbox.org/permlink/wRC1vY7v1gZYobjX). I wonder whether there is already a bug report. And oh boy, the problem disappears under ptrace. Nice. – sehe Jun 23 '20 at 21:20
  • @frank I have to conclude it must never have actually worked with std::unordered_map https://i.imgur.comI/BUPYfMK.png this was tested using a simple test script + docker http://stackoverflow-sehe.s3.amazonaws.com/5e1122ee-9e29-4040-8be3-25f2222b7941/container-tests.zip if you want to repeat. – sehe Jun 26 '20 at 22:10