c++ boost interprocess exchange (copy) non-shared and shared string vector

Question

I want to achieve what was achieved in the answer to this question, but for a vector of strings i.e. string objects. I want to be able to write common code that can be used with either heap memory or shared memory. I have emulated the code and wrote a class to implement a vector of strings that would work for with both heap memory or shared memory. I got it working for the most part. However, when I try to initialize/create a shared memory vector of strings from a heap allocated vector of strings I am getting compile errors. In the question referenced above, shared memory vector was initialized from non-shared memory vector (to quote "this works because of ... MAGIC!"). Not sure if its because that question was dealing with a vector of ints which are easily copyable and movable. I have provided the copy constructor for the shared memory string class but it still doesn't work. I've provided my code below (commented out the code that's giving me compilation errors). Any help is appreciated. Thanks.

#include <boost/interprocess/managed_shared_memory.hpp>
#include <boost/interprocess/allocators/allocator.hpp>
#include <boost/interprocess/containers/vector.hpp>
#include <boost/interprocess/containers/string.hpp>
#include <boost/container/scoped_allocator.hpp>
#include <iostream>
#include <string>
#include <sstream>

namespace bip = boost::interprocess;

namespace generic
{

template <typename T, typename Alloc/* = std::allocator<T>*/ >
using vector = bip::vector<T, typename Alloc::template rebind<T>::other >;

template <typename Alloc/* = std::allocator<T>*/ >
using GenericString = bip::basic_string<char, std::char_traits<char>
                           , typename Alloc::template rebind<char>::other>;

template <typename Alloc>
class String
{
public:
  typedef Alloc allocator_type; // ties in with uses_allocator/scoped_allocator

  String(Alloc alloc = Alloc())
  : mString(alloc)
  {
  }

  String(const std::string str, Alloc alloc = Alloc())
  : mString(str.c_str(), alloc)
  {
  }

  String(const char* str, Alloc alloc = Alloc())
  : mString(str, alloc)
  {
  }

  template <typename OtherAlloc>
  String(String<OtherAlloc> const& other, Alloc alloc = Alloc())
  : mString(other.mString.begin(), other.mString.end(), alloc)
  {
  }

  std::string ToString() { return std::string(mString.begin(), mString.end()); }

  friend std::ostream& operator<<(std::ostream& os, const String& str)
  {
    os << str.mString;
    return os;
  }

private:
  template<typename OtherAlloc> friend struct String;
  GenericString<Alloc> mString;
};

template <typename Alloc>
class StringVector
{
public:
  typedef Alloc allocator_type; // ties in with uses_allocator/scoped_allocator
  typedef vector<String<Alloc>, Alloc> GenericStringVector;
  using Iterator = typename GenericStringVector::iterator;
  using ConstIterator = typename GenericStringVector::const_iterator;

  StringVector(Alloc alloc = Alloc())
  : mStrings(alloc)
  {
  }

  template <typename OtherAlloc>
  StringVector(StringVector<OtherAlloc> const& other, Alloc alloc = Alloc())
  : mStrings(other.mStrings.begin(), other.mStrings.end(), alloc)
  {
  }

  void Add(const String<Alloc> str) { mStrings.emplace_back(str); }

  void Remove(Iterator iter) { mStrings.erase(iter); }

  void Clear() { mStrings.clear(); }

  void Print() const
  {
    std::cout << "[";
    for (ConstIterator it = mStrings.begin(); it != mStrings.end(); it++)
      std::cout << (it == mStrings.begin() ? "" : ", ") << *it;
    std::cout << "]\n";
  }

private:
  template<typename OtherAlloc> friend struct StringVector;
  GenericStringVector mStrings;
};

}

namespace heap
{
  using VAlloc = std::allocator<void>;
  using String = generic::String<VAlloc>;
  using StringVector = generic::StringVector<VAlloc>;
}

namespace shared
{
  using VAlloc =boost::container::scoped_allocator_adaptor<
          bip::allocator<void, bip::managed_shared_memory::segment_manager> >;

  using String = generic::String<VAlloc>;
  using StringVector = generic::StringVector<VAlloc>;
}

int main(void)
{
  struct shm_remove {
      shm_remove()  { bip::shared_memory_object::remove("MySharedMemory"); }
      ~shm_remove() { bip::shared_memory_object::remove("MySharedMemory"); }
  } remover;

  std::cout << "Heap based storage: \n";
  heap::StringVector svec;

  svec.Add(heap::String("test1"));
  svec.Add(heap::String("test2"));
  svec.Add(heap::String("test3"));
  svec.Add(heap::String("test4"));
  svec.Print();

  std::cout << "Shared memory storage: \n";
  bip::managed_shared_memory seg(bip::create_only, "MySharedMemory", 65536);
  shared::VAlloc shalloc(seg.get_segment_manager());

  shared::StringVector sh_svec(shalloc);
  sh_svec.Add(shared::String("test1", shalloc));
  sh_svec.Add(shared::String("test2", shalloc));
  sh_svec.Add(shared::String("test3", shalloc));
  sh_svec.Add(shared::String("test4", shalloc));
  sh_svec.Print();


  shared::StringVector sh_svec2(sh_svec, shalloc);
  sh_svec2.Print();

  //shared::StringVector sh_svec3(svec, shalloc);   // gives compile error :(
  //sh_svec3.Print();
}

Including compilation errors:

In file included from src/test_string_vector.cc:5:0:
/usr/include/boost/container/scoped_allocator.hpp: In instantiation of ‘boost::container::container_detail::scoped_allocator_adaptor_base<OuterAlloc>::scoped_allocator_adaptor_base() [with OuterAlloc = boost::interprocess::allocator<void, boost::interprocess::segment_manager<char, boost::interprocess::rbtree_best_fit<boost::interprocess::mutex_family>, boost::interprocess::iset_index> >]’:
/usr/include/boost/container/scoped_allocator.hpp:963:7:   required from ‘boost::container::scoped_allocator_adaptor<OuterAlloc, InnerAllocs>::scoped_allocator_adaptor() [with OuterAlloc = boost::interprocess::allocator<void, boost::interprocess::segment_manager<char, boost::interprocess::rbtree_best_fit<boost::interprocess::mutex_family>, boost::interprocess::iset_index> >; InnerAllocs = {}]’
/usr/include/boost/container/vector.hpp:1174:15:   required from ‘void boost::container::vector<T, Allocator>::assign(FwdIt, FwdIt, typename boost::container::container_detail::enable_if_c<((! boost::container::container_detail::is_convertible<InIt, typename boost::container::allocator_traits<Allocator>::size_type>::value) && ((! boost::container::container_detail::is_input_iterator<InIt>::value) && (! boost::move_detail::is_same<typename boost::container::container_detail::version<Allocator>::type, boost::container::container_detail::integral_constant<unsigned int, 0u> >::value)))>::type*) [with FwdIt = boost::container::container_detail::vec_iterator<generic::String<std::allocator<void> >*, true>; T = generic::String<boost::container::scoped_allocator_adaptor<boost::interprocess::allocator<void, boost::interprocess::segment_manager<char, boost::interprocess::rbtree_best_fit<boost::interprocess::mutex_family>, boost::interprocess::iset_index> > > >; Allocator = boost::container::scoped_allocator_adaptor<boost::interprocess::allocator<generic::String<boost::container::scoped_allocator_adaptor<boost::interprocess::allocator<void, boost::interprocess::segment_manager<char, boost::interprocess::rbtree_best_fit<boost::interprocess::mutex_family>, boost::interprocess::iset_index> > > >, boost::interprocess::segment_manager<char, boost::interprocess::rbtree_best_fit<boost::interprocess::mutex_family>, boost::interprocess::iset_index> > >; typename boost::container::container_detail::enable_if_c<((! boost::container::container_detail::is_convertible<InIt, typename boost::container::allocator_traits<Allocator>::size_type>::value) && ((! boost::container::container_detail::is_input_iterator<InIt>::value) && (! boost::move_detail::is_same<typename boost::container::container_detail::version<Allocator>::type, boost::container::container_detail::integral_constant<unsigned int, 0u> >::value)))>::type = void]’
/usr/include/boost/container/vector.hpp:864:7:   required from ‘boost::container::vector<T, Allocator>::vector(InIt, InIt, const allocator_type&) [with InIt = boost::container::container_detail::vec_iterator<generic::String<std::allocator<void> >*, true>; T = generic::String<boost::container::scoped_allocator_adaptor<boost::interprocess::allocator<void, boost::interprocess::segment_manager<char, boost::interprocess::rbtree_best_fit<boost::interprocess::mutex_family>, boost::interprocess::iset_index> > > >; Allocator = boost::container::scoped_allocator_adaptor<boost::interprocess::allocator<generic::String<boost::container::scoped_allocator_adaptor<boost::interprocess::allocator<void, boost::interprocess::segment_manager<char, boost::interprocess::rbtree_best_fit<boost::interprocess::mutex_family>, boost::interprocess::iset_index> > > >, boost::interprocess::segment_manager<char, boost::interprocess::rbtree_best_fit<boost::interprocess::mutex_family>, boost::interprocess::iset_index> > >; boost::container::vector<T, Allocator>::allocator_type = boost::container::scoped_allocator_adaptor<boost::interprocess::allocator<generic::String<boost::container::scoped_allocator_adaptor<boost::interprocess::allocator<void, boost::interprocess::segment_manager<char, boost::interprocess::rbtree_best_fit<boost::interprocess::mutex_family>, boost::interprocess::iset_index> > > >, boost::interprocess::segment_manager<char, boost::interprocess::rbtree_best_fit<boost::interprocess::mutex_family>, boost::interprocess::iset_index> > >]’
src/test_string_vector.cc:78:65:   required from ‘generic::StringVector<Alloc>::StringVector(const generic::StringVector<OtherAlloc>&, Alloc) [with OtherAlloc = std::allocator<void>; Alloc = boost::container::scoped_allocator_adaptor<boost::interprocess::allocator<void, boost::interprocess::segment_manager<char, boost::interprocess::rbtree_best_fit<boost::interprocess::mutex_family>, boost::interprocess::iset_index> > >]’
src/test_string_vector.cc:151:46:   required from here
/usr/include/boost/container/scoped_allocator.hpp:762:7: error: no matching function for call to ‘boost::interprocess::allocator<void, boost::interprocess::segment_manager<char, boost::interprocess::rbtree_best_fit<boost::interprocess::mutex_family>, boost::interprocess::iset_index> >::allocator()’
       {}
       ^
In file included from /usr/include/boost/interprocess/segment_manager.hpp:38:0,
                 from /usr/include/boost/interprocess/detail/managed_memory_impl.hpp:30,
                 from /usr/include/boost/interprocess/managed_shared_memory.hpp:25,
                 from src/test_string_vector.cc:1:
/usr/include/boost/interprocess/allocators/allocator.hpp:142:4: note: candidate: template<class T2> boost::interprocess::allocator<T, SegmentManager>::allocator(const boost::interprocess::allocator<T2, SegmentManager>&)
    allocator(const allocator<T2, SegmentManager> &other)
    ^
/usr/include/boost/interprocess/allocators/allocator.hpp:142:4: note:   template argument deduction/substitution failed:
In file included from src/test_string_vector.cc:5:0:
/usr/include/boost/container/scoped_allocator.hpp:762:7: note:   candidate expects 1 argument, 0 provided
       {}
       ^
In file included from /usr/include/boost/interprocess/segment_manager.hpp:38:0,
                 from /usr/include/boost/interprocess/detail/managed_memory_impl.hpp:30,
                 from /usr/include/boost/interprocess/managed_shared_memory.hpp:25,
                 from src/test_string_vector.cc:1:
/usr/include/boost/interprocess/allocators/allocator.hpp:136:4: note: candidate: boost::interprocess::allocator<T, SegmentManager>::allocator(const boost::interprocess::allocator<T, SegmentManager>&) [with T = void; SegmentManager = boost::interprocess::segment_manager<char, boost::interprocess::rbtree_best_fit<boost::interprocess::mutex_family>, boost::interprocess::iset_index>]
    allocator(const allocator &other)
    ^
/usr/include/boost/interprocess/allocators/allocator.hpp:136:4: note:   candidate expects 1 argument, 0 provided
/usr/include/boost/interprocess/allocators/allocator.hpp:131:4: note: candidate: boost::interprocess::allocator<T, SegmentManager>::allocator(boost::interprocess::allocator<T, SegmentManager>::segment_manager*) [with T = void; SegmentManager = boost::interprocess::segment_manager<char, boost::interprocess::rbtree_best_fit<boost::interprocess::mutex_family>, boost::interprocess::iset_index>; boost::interprocess::allocator<T, SegmentManager>::segment_manager = boost::interprocess::segment_manager<char, boost::interprocess::rbtree_best_fit<boost::interprocess::mutex_family>, boost::interprocess::iset_index>]
    allocator(segment_manager *segment_mngr)
    ^
/usr/include/boost/interprocess/allocators/allocator.hpp:131:4: note:   candidate expects 1 argument, 0 provided
Makefile:16: recipe for target 'obj/test_string_vector.o' failed
make: *** [obj/test_string_vector.o] Error 1

Marcus Brody · Accepted Answer · 2017-06-05T23:45:40.367

I think I understand your intent now . you want this piece of code where OtherAlloc = std::allocator , Alloc = bi:allocator

template <typename OtherAlloc>
  StringVector(StringVector<OtherAlloc> const& other, Alloc alloc = Alloc())
  : mStrings(other.mStrings.begin(), other.mStrings.end(), alloc)
  {
  }

to call a copy constructor you provided to convert from heap to shared type

 template <typename OtherAlloc>
  String(String<OtherAlloc> const& other, Alloc alloc = Alloc())
  : mString(other.mString.begin(), other.mString.end(), alloc)
  {
  }

But boost::interprocess::vector::assign(first,last) does not pass instance of your allocator , and it looks like it uses Alloc alloc = Alloc() where Alloc = boost::interprocess::allocator and it does not have default constructor. It correlates to all my posts :-) I will try writing my code tomorrow , very challenging puzzle.

I have modified your code today please see below and see if you like it.

#include <boost/interprocess/managed_shared_memory.hpp>
#include <boost/interprocess/allocators/allocator.hpp>
#include <boost/interprocess/containers/vector.hpp>
#include <boost/interprocess/containers/string.hpp>
#include <boost/container/scoped_allocator.hpp>
#include <iostream>
#include <string>
#include <sstream>
#include <algorithm>

namespace bip = boost::interprocess;

namespace generic
{

template <typename T, typename Alloc/* = std::allocator<T>*/ >
using vector = bip::vector<T, typename Alloc::template rebind<T>::other >;

template <typename Alloc/* = std::allocator<T>*/ >
using GenericString = bip::basic_string<char, std::char_traits<char>
                           , typename Alloc::template rebind<char>::other>;

template <typename Alloc>
class String
{
public:
  typedef Alloc allocator_type; // ties in with uses_allocator/scoped_allocator

  template<typename ...T>
  String(const std::string &str, T && ...t)
  : alloc_(std::forward<T>(t)...) , mString(str.c_str(), alloc_)
  {}

  template<typename ...T>
  String(const char* str, T && ...t)
  : alloc_(std::forward<T>(t)...) , mString(str, alloc_)
  {}

  template <typename OtherAlloc, typename ...T>
  String(String<OtherAlloc> const& other, T && ...t)
  : alloc_(std::forward<T>(t)...) , mString(other.mString.begin(), other.mString.end(), alloc_)
  {}

  std::string ToString() { return std::string(mString.begin(), mString.end()); }

  friend std::ostream& operator<<(std::ostream& os, const String& str)
  {
    os << str.mString;
    return os;
  }

private:
  template<typename OtherAlloc> friend struct String;
  Alloc alloc_;
  GenericString<Alloc> mString;
};

template <typename Alloc>
class StringVector
{
public:
  typedef Alloc allocator_type; // ties in with uses_allocator/scoped_allocator
  typedef vector<String<Alloc>, Alloc> GenericStringVector;
  using Iterator = typename GenericStringVector::iterator;
  using ConstIterator = typename GenericStringVector::const_iterator;

  StringVector()
  : alloc_(), mStrings(alloc_)
  {}

  template<typename T>
  StringVector(T && t)
  : alloc_(std::forward<T>(t)), mStrings(alloc_)
  {}

  template <typename OtherAlloc, typename ...T>
  StringVector(StringVector<OtherAlloc> const& other, T && ...t)
  : alloc_(std::forward<T>(t)...), mStrings(alloc_)
  {
      std::transform ( other.mStrings.begin(), other.mStrings.end(), std::back_inserter(mStrings), [&t...](auto &s) {
          return typename GenericStringVector::value_type(s, std::forward<T>(t)...);
      });
  }

  void Add(const String<Alloc> str) { mStrings.emplace_back(str); }

  void Remove(Iterator iter) { mStrings.erase(iter); }

  void Clear() { mStrings.clear(); }

  void Print() const
  {
    std::cout << "[";
    for (ConstIterator it = mStrings.begin(); it != mStrings.end(); it++)
      std::cout << (it == mStrings.begin() ? "" : ", ") << *it;
    std::cout << "]\n";
  }

private:
  template<typename OtherAlloc> friend struct StringVector;
  Alloc alloc_;
  GenericStringVector mStrings;
};

}

namespace heap
{
  using VAlloc = std::allocator<void>;
  using String = generic::String<VAlloc>;
  using StringVector = generic::StringVector<VAlloc>;
}

namespace shared
{
  using VAlloc =boost::container::scoped_allocator_adaptor<
          bip::allocator<char, bip::managed_shared_memory::segment_manager> >;

  using String = generic::String<VAlloc>;
  using StringVector = generic::StringVector<VAlloc>;
}

int main(void)
{
  struct shm_remove {
      shm_remove()  { bip::shared_memory_object::remove("MySharedMemory"); }
      ~shm_remove() { bip::shared_memory_object::remove("MySharedMemory"); }
  } remover;

  std::cout << "Heap based storage: \n";
  heap::StringVector svec;

  svec.Add(heap::String("test1"));
  svec.Add(heap::String("test2"));
  svec.Add(heap::String("test3"));
  svec.Add(heap::String("test4"));
  svec.Print();

  std::cout << "Shared memory storage: \n";
  bip::managed_shared_memory seg(bip::create_only, "MySharedMemory", 65536);
  auto smp = seg.get_segment_manager();

  shared::StringVector sh_svec(smp);
  sh_svec.Add(shared::String("test1", smp));
  sh_svec.Add(shared::String("test2", smp));
  sh_svec.Add(shared::String("test3", smp));
  sh_svec.Add(shared::String("test4", smp));
  sh_svec.Print();


  shared::StringVector sh_svec2(sh_svec, smp);
  sh_svec2.Print();

  shared::StringVector sh_svec3(svec, smp);   // does not give compile error 
  sh_svec3.Print();
}

yeah. I have a need where my code needs to work both in heap memory and shared memory memory systems. Hence I am trying to come up with basic data structures which will allow me to this. Thanks for taking the time. — Dnj Abc, Jun 05 '17 at 18:47
No worries I have re-factored your generic code slightly, it compiles it runs. — Marcus Brody, Jun 05 '17 at 23:47
This is awesome. Thank you. One last favor. Could you please explain what you did in the StringVector copy constructor. What you did seems like black magic to me. :) — Dnj Abc, Jun 06 '17 at 02:14
So, because std::allocator() constructor does not require parameter but boost::interprocess::allocator(segment_manager *) requires at least one parameter pointer to segment_manager to achieve same constuctor for both I used variadic template for StringVector copy constructor. — Marcus Brody, Jun 06 '17 at 20:08
**T && ...t** gets converted to 0...N parameter in our case 0 or 1 . I don't pass allocator , but I pass parameters to allocator. For Heap pass nothing , for Shared pass segment_manager *. Tried longer comment it did not take it :-( — Marcus Brody, Jun 06 '17 at 20:28
Thanks. The use of lambda and variadics was clever. Hope you don't have any issues if I publish this on my personal github account later. Hopefully it'll be useful to others. Thanks again. — Dnj Abc, Jun 06 '17 at 21:58

score 0 · Answer 2 · 2017-06-05T07:02:42.330

0

You did not provide your compilation errors, however just by briefly looking at the code and some working knowlege of boost::interprocess , I think your problem lies in std::allocator used for boost::interprocess , but I am not 100% sure, so please provide compilation errors. In the meantime if you like
you can look at our open-source stack solution for Shared, Mmap, Heap memory_types.hpp I think you can use boost::interprocess::allocator with boost::interprocess::managed_heap_memory::segment_manager or something like that. Please also specify if you use C++03 or C++11 compiler as I think in C++03 it was typedef T* pointer_type and that will not work with boost shared memory as it needs offset_ptr , basically offset from the beginning of memory segment and not an absolute address.

I modified your code you had used shared::StringVector to store SHM structure So now it compiles:

heap::StringVector sh_svec3(svec);   // does not gives compile error :-)
sh_svec3.Print();

edited Jun 05 '17 at 07:02

answered Jun 05 '17 at 01:35

Added the compilation errors. Didn't want to make the question too long... :( – Dnj Abc Jun 05 '17 at 02:17
It compiles and links for me , I am using gcc 5.4 and boost 1.61 – Jun 05 '17 at 04:20
I assume you are using C++11 because otherwise **template using** would not compile, so the only question is your boost version < 1.58 – Jun 05 '17 at 04:27
did you uncomment the last two lines the end... //shared::StringVector sh_svec3(svec, shalloc); // gives compile error :( //sh_svec3.Print(); I commented them to test other parts of the code. Sorry about that. FWIW, I compiled it with boost 1.58. – Dnj Abc Jun 05 '17 at 05:42
got it . boost::interprocess::allocator does not have default contructor as in your generic code `String(Alloc alloc = Alloc()) : mString(alloc) { }` so the only way to solve it is as I said in my answer , look at the samples in our project and use boost::interprocess::managed_heap_memory::segment_manager – Jun 05 '17 at 05:56
did not follow :( – Dnj Abc Jun 05 '17 at 06:02
Alloc() - does not exist in boost::interprocess::allocator , boost::interprocess::allocator(segment_manager *) takes 1 argument , so you will have to slightly change your code when you use `Alloc alloc = Alloc()` – Jun 05 '17 at 06:12
The default allocator is for the heap allocator. I am trying to generalize the code in my generic namespace so that I can use it with both heap memory and shared memory by passing different allocators (and defaulting for heap). When creating shared memory objects I am always passing the shared memory allocator. Still a little confused... :( – Dnj Abc Jun 05 '17 at 06:31
I think I found your problem , see my updated response , the last 2 lines are wrong should be : `heap::StringVector sh_svec3(svec); // does not gives compile error :-) sh_svec3.Print();` – Jun 05 '17 at 07:06
I think you misread my intent. I want to create a shared memory vector from a std::vector (i.e heap based string vector). Since I have a copy constructor that accepts a StringVector, I was hoping that would be sufficient to get it working. If you look at the example link I referred to in the beginning of my post, that example does exact that. However, that example is dealing with a vector ints and I trying to achieve a similar thing with a vector strings. – Dnj Abc Jun 05 '17 at 07:45

c++ boost interprocess exchange (copy) non-shared and shared string vector

2 Answers2