1

I am looking for a library which allows to get a circular buffer on disk.
In Boost there is something similar, but it is an in memory based container: circular_buffer.

Pietro
  • 12,086
  • 26
  • 100
  • 193
  • Maybe I'm the only one...but what do you mean by a *circular buffer on disk*. Do you mean the buffer is *only* on the disk? A buffer of file handles? A circular buffer that is partially paged onto a disk?!? – Rollen Mar 25 '15 at 14:39
  • @RollenD'Souza - I mean a file that can grow up to a certain limit, and when this limit is passed, it starts overwriting itself from the beginning. A typical example is a log file. – Pietro Mar 25 '15 at 17:13
  • @RollenD'Souza - Anyway you are probably right: it may not be called circular buffer when it is on disk... – Pietro Mar 25 '15 at 17:15
  • 1
    @Pietro: Normally circular log files go back and forth between two log files, so you always have at least XMb of logs on disk, and the oldest entry is always at the top of the file. I don't think boost can trivially do that. – Mooing Duck Mar 25 '15 at 20:33

1 Answers1

7

You can call it whatever you think is natural.

You're looking for memory mapped files.

Using the right allocator, you can make containers be allocating in this memory mapped region. That would make the container "on disk".

I'll see whether Boost Circularbuffer supports this directly.

Update Yes.

The best thing is, this gives you full possibility to even use IPC synchronization and thread synchronization. Using a "private" memory map you could map the buffer read-writable without writing changes back to disk in some of the processes.

Proof of concept:

Live On Coliru ¹

#include <boost/circular_buffer.hpp>
#include <boost/interprocess/allocators/allocator.hpp>
#include <boost/interprocess/managed_mapped_file.hpp>

namespace bip = boost::interprocess;

struct message {
    int data[32];
};

int main()
{
    bip::managed_mapped_file mmf(bip::open_or_create, "/tmp/circ_buffer.bin", 4ul << 20);
    typedef bip::allocator<message, bip::managed_mapped_file::segment_manager> allocator;

    boost::circular_buffer<message, allocator> instance(100, mmf.get_segment_manager());
}

¹ On Coliru the filesize is - understandably constrained.

sehe
  • 374,641
  • 47
  • 450
  • 633
  • Added a proof of concept with Live demo – sehe Mar 25 '15 at 20:27
  • In my case I cannot have the file mapped in memory, but it is an interesting solution. – Pietro Mar 31 '15 at 08:58
  • Because of the size of the file. And being just for logs, it is not "valuable" enough to dedicate any RAM to it. – Pietro Apr 01 '15 at 09:06
  • 1
    @Pietro This is why I asked. It is a very common misconception that memory mapped files take RAM. It's only virtual. In fact, memory mapped files are precisely the _solution_ to use when your datastructures cannot fit into physical RAM. – sehe Apr 01 '15 at 09:08
  • 1
    @Pietro see e.g. [this answer](http://stackoverflow.com/a/23384879/85371) that addresses how memory mapping works, in general, and the follow-up [Memory usage when using mapped_file](http://stackoverflow.com/a/23468466/85371) or – sehe Apr 01 '15 at 09:20
  • What if the messages are not with the same size as you in demo, can we have a serialize function for the class which give us a variable length string ? – prehistoricpenguin Aug 23 '19 at 02:33