I wish to share objects containing internal pointers between unrelated processes on Linux.
Similar to this question?
Does boost.interprocess support this?
Any data structure containing raw pointers cannot be used for IPC unless each process is somehow able to map the memory to the same location.
What is boost's answer to this?
If the address space is shared it is safe to use such an allocator for the complete structure
You're mixing "shared address space" and "address mapped at the same address". Because if there was no sharing of address space, there would also not be an issue storing raw pointers.
What is important is to use fancy pointers like boost::interprocess::offset_ptr<T>. The interprocess allocator uses offset_ptr<T>.
This is why you need to use container implementations that support fancy pointers.
You're very close. The canonical way to do what you sketch would look like this:
#include <boost/container/string.hpp>
#include <boost/container/vector.hpp>
#include <boost/interprocess/allocators/allocator.hpp>
#include <boost/interprocess/managed_shared_memory.hpp>
#include <boost/interprocess/managed_mapped_file.hpp>
#include <string>
namespace bip = boost::interprocess;
namespace Shared {
#ifdef COLIRU
using Segment = bip::managed_mapped_file;
#else
using Segment = bip::managed_shared_memory;
#endif
using Mgr = Segment::segment_manager;
template <typename T> using Alloc = bip::allocator<T, Mgr>;
template <typename T> using Vector = std::vector<T, Alloc<T>>;
using String = boost::container::basic_string< //
char, std::char_traits<char>, Alloc<char>>;
struct Bar {
using allocator_type = Alloc<char>;
String first_name, last_name;
template <typename Alloc> Bar(std::string_view first_name, std::string_view last_name, Alloc alloc) :
first_name(first_name, alloc), last_name(last_name, alloc) {}
};
struct Snafu {
std::array<int, 5> no_problem;
};
struct Foo {
Vector<Bar> bars;
Vector<Snafu> snafus;
template <typename Alloc> //
Foo(Alloc alloc) : bars(alloc)
, snafus(alloc) {}
};
}
#include <iostream>
static inline std::ostream& operator<<(std::ostream& os, Shared::Foo const& foo) {
os << "Foo\n================\nBars:\n";
for (auto& [f, l] : foo.bars)
os << " - " << f << ", " << l << "\n";
os << "nafus:";
for (auto& s : foo.snafus) {
os << "\n - ";
for (auto el : s.no_problem)
os << " " << el;
}
return os << "\n";
}
int main() { Shared::Segment msm(bip::open_or_create, "my_shared_mem", 10ull << 10);
Shared::Foo& foo = *msm.find_or_construct<Shared::Foo>("the_foo") //
(msm.get_segment_manager()); // constructor arguments
foo.bars.emplace_back("John", "Doe", msm.get_segment_manager());
foo.bars.emplace_back("Jane", "Deer", msm.get_segment_manager());
foo.bars.emplace_back("Igor", "Stravinsky", msm.get_segment_manager());
foo.bars.emplace_back("Rimsky", "Korsakov", msm.get_segment_manager());
foo.snafus.push_back({1, 2, 3});
foo.snafus.push_back({2, 3, 4});
foo.snafus.push_back({3, 4, 5, 6, 7});
std::cout << foo << "\n";
}
Output first run:
Foo
================
Bars:
- John, Doe
- Jane, Deer
- Igor, Stravinsky
- Rimsky, Korsakov
nafus:
- 1 2 3 0 0
- 2 3 4 0 0
- 3 4 5 6 7
Output second run:
Foo
================
Bars:
- John, Doe
- Jane, Deer
- Igor, Stravinsky
- Rimsky, Korsakov
- John, Doe
- Jane, Deer
- Igor, Stravinsky
- Rimsky, Korsakov
nafus:
- 1 2 3 0 0
- 2 3 4 0 0
- 3 4 5 6 7
- 1 2 3 0 0
- 2 3 4 0 0
- 3 4 5 6 7
As always, I cannot be remiss about scoped_allocator_adaptor, the best thing since sliced semiconductors:
template <typename T>
using Alloc = boost::container::scoped_allocator_adaptor< //
bip::allocator<T, Mgr>>;
Now, anywhere uses_allocator protocol is used, you get magic allocator propagation:
#include <boost/container/scoped_allocator.hpp>
#include <boost/container/string.hpp>
#include <boost/container/vector.hpp>
#include <boost/interprocess/allocators/allocator.hpp>
#include <boost/interprocess/managed_mapped_file.hpp>
#include <boost/interprocess/managed_shared_memory.hpp>
#include <string>
namespace bip = boost::interprocess;
namespace Shared {
#ifdef COLIRU
using Segment = bip::managed_mapped_file;
#else
using Segment = bip::managed_shared_memory;
#endif
using Mgr = Segment::segment_manager;
template <typename T>
using Alloc = boost::container::scoped_allocator_adaptor< //
bip::allocator<T, Mgr>>;
template <typename T> using Vector = std::vector<T, Alloc<T>>;
using String = boost::container::basic_string< //
char, std::char_traits<char>, Alloc<char>>;
struct Bar {
using allocator_type = Alloc<char>;
String first_name, last_name;
template <typename Alloc>
Bar(std::allocator_arg_t, Alloc alloc) : first_name(alloc)
, last_name(alloc) {}
template <typename Alloc>
Bar(Alloc&& alloc) : first_name(alloc)
, last_name(alloc) {}
Bar(Bar const&) = default;
template <typename Alloc>
Bar(Bar const& rhs, Alloc alloc) : first_name(rhs.first_name, alloc), last_name(rhs.last_name, alloc) {}
Bar& operator=(Bar const&) = default;
template <typename Alloc> Bar(std::string_view first_name, std::string_view last_name, Alloc alloc) :
first_name(first_name, alloc), last_name(last_name, alloc) {}
};
struct Snafu {
std::array<int, 5> no_problem;
};
struct Foo {
Vector<Bar> bars;
Vector<Snafu> snafus;
template <typename Alloc> //
Foo(Alloc alloc) : bars(alloc)
, snafus(alloc) {}
};
}
#include <iostream>
static inline std::ostream& operator<<(std::ostream& os, Shared::Foo const& foo) {
os << "Foo\n================\nBars:\n";
for (auto& [f, l] : foo.bars)
os << " - " << f << ", " << l << "\n";
os << "Snafus:";
for (auto& s : foo.snafus) {
os << "\n - ";
for (auto el : s.no_problem)
os << " " << el;
}
return os << "\n";
}
int main() { Shared::Segment msm(bip::open_or_create, "my_shared_mem", 10ull << 10);
Shared::Foo& foo = *msm.find_or_construct<Shared::Foo>("the_foo") //
(msm.get_segment_manager()); // constructor arguments
foo.bars.emplace_back("John", "Doe");
foo.bars.emplace_back("Jane", "Deer");
foo.bars.emplace_back("Igor", "Stravinsky");
foo.bars.emplace_back("Rimsky", "Korsakov");
foo.snafus.push_back({1, 2, 3});
foo.snafus.push_back({2, 3, 4});
foo.snafus.push_back({3, 4, 5, 6, 7});
std::cout << foo << "\n";
}
With identical output
The answer according to the documentation is no.
Limitations When Constructing Objects In Mapped Regions
Offset pointers instead of raw pointers References forbidden Virtuality forbidden Be careful with static class members
When two processes create a mapped region of the same mappable object, two processes can communicate writing and reading that memory. A process could construct a C++ object in that memory so that the second process can use it. However, a mapped region shared by multiple processes, can't hold any C++ object, because not every class is ready to be a process-shared object, specially, if the mapped region is mapped in different address in each process.
If you could get it to work with boost.interprocess it probably wouldn't be portable.
Boost does provide an alternative solution. There are several pieces of magic which make this work:
Boost provides containers which use smart offset pointers (boost::interprocess::offset_ptr) which can be used in place of regular STL containers.
https://stackoverflow.com/a/7159729/1569204 has a concise explanation of how this type of pointer can be implemented (not boost specific):
template<class T> class offset_ptr {
size_t offset; public:
T* operator->() { return reinterpret_cast<T*>(reinterpret_cast<char*>(this)+offset); } };
This is very clever. I'm not sure why this has not yet made it into the ISO standard (perhaps it has and I am just ignorant).
See https://en.cppreference.com/w/cpp/named_req/Allocator#Fancy_pointers
When the member type pointer is not a raw pointer type, it is commonly referred to as a "fancy pointer". Such pointers were introduced to support segmented memory architectures and are used today to access objects allocated in address spaces that differ from the homogeneous virtual address space that is accessed by raw pointers. An example of a fancy pointer is the mapping address-independent pointer boost::interprocess::offset_ptr, which makes it possible to allocate node-based data structures such as std::set in shared memory and memory mapped files mapped in different addresses in every process. Fancy pointers can be used independently of the allocator that provided them, through the class template std::pointer_traits (since C++11). The function std::to_address can be used to obtain a raw pointer from a fancy pointer. (since C++20)
Use of fancy pointers and customized size/different type in the standard libary are conditionally supported. Implementations may require that member type pointer, const_pointer, size_type, and difference_type are value_type*, const value_type*, std::size_t, and std::ptrdiff_t, respectively.
There is a good explanation here:
The boost version of the containers uses the pointer type specified by the allocator. This is why allocator_traits includes a pointer type. In theory an STL implementation (post C++11) should use it instead of assuming a raw pointer. Boosts containers ensure that this is done. So the boost allocator not only returns offset_ptr pointers but informs the container template that it should use them as well.
The other bit of magic is a standard way of getting a template to use the allocator of the structure containing it. See What is "uses allocator" and "scoped allocator" construction in c++
This bit of magic has been in C++ since at least C++11.