std::fstream with multiple buffers?

Question

You can specify one buffer for your file stream like that:

char buf[BUFFER_SIZE];

std::ofstream file("file", std::ios_base::binary | std::ios_base::out);
if (file.is_open())
{
    file.rdbuf()->pubsetbuf(buf, BUFFER_SIZE);
    file << "abcd";
}

What I want to do now, is using more than just one buffer:

char* buf[] = { new char[BUFFER_SIZE], new char[BUFFER_SIZE], new char[BUFFER_SIZE], };

Is it possible without creating a custom derivation of std::streambuf?

EDIT: I think I need to explain what I want to do in more detail. Please consider the following situation: - The file(s) I want to read won't fit into memory - The file while be accessed by some kind of a binary jump search

So, if you split the file into logical pages of a specific size, then I would like to provide multiple buffers which are representing specific pages. This would increase performance when a file location is read and the related page is already in a buffer.

1) I want to specify a huge buffer and don't want to force the system to allocate one piece of that size. 2) I'm having multiple huge files which are frequently randomly accessed by the application. So, I want to read and write into a buffer to increase performance. — 0xbadf00d, Jun 09 '11 at 14:43

score 1 · Answer 1 · answered Jun 09 '11 at 18:32

1

I gather from the comment that you want to do a kind of scatter-gather I/O. I'm pretty sure there's no support for that in the C++ standard I/O streams library, so you'll have to roll your own.

If you want to do this efficiently, you can use OS support for scatter-gather. E.g., POSIX/Unix-like systems have writev for this purpose.

answered Jun 09 '11 at 18:32

Fred Foo

355,277
75
744
836

Thanks for your answer. I've updated my initial post and described my requirements in a little bit more detail. – 0xbadf00d Jun 10 '11 at 17:08

score 1 · Answer 2 · answered Jun 10 '11 at 17:56

1

There's nothing like this provided by Standard. However, depending on your platform, you can use Memory Mapped Files, which provide the same functionality. Windows and Linux both provide them.

answered Jun 10 '11 at 17:56

Puppy

144,682
38
256
465

1

This seems to match the OP's requirements, +1. Don't forget the [portable memory mapped files in Boost](http://www.boost.org/doc/libs/release/libs/iostreams/doc/classes/mapped_file.html). – Fred Foo Jun 10 '11 at 20:30
I will take a look at that, but I already implemented a quite good looking solution (from my point of view). I would appreciate if you share your thoughts on that. – 0xbadf00d Jun 11 '11 at 09:56

score 1 · Accepted Answer · answered Jun 11 '11 at 09:32

I will take a look at boost::iostreams::mapped_file, but I think my requirement is much simpler. I've created a custom class derived from basic_filebuf.

template<typename char_type>
class basic_filemultibuf : public std::basic_filebuf<char_type/*, std::char_traits<char_type>*/>
{
private:
    char_type**     m_buffers;
    std::ptrdiff_t  m_buffer_count,
                    m_curent_buffer;
    std::streamsize m_buffer_size;

protected:
    virtual int_type overflow(int_type meta = traits_type::eof())
    {
        if (this->m_buffer_count > 0)
        {
            if (this->m_curent_buffer == this->m_buffer_count)
                this->m_curent_buffer = 0;
            this->basic_filebuf::setbuf(this->m_buffers[this->m_curent_buffer++], this->m_buffer_size);
        }

        return this->basic_filebuf::overflow(meta);
    }

public:
    basic_filemultibuf(basic_filebuf const& other)
        : basic_filebuf(other),
          m_buffers(NULL),
          m_buffer_count(0),
          m_curent_buffer(-1),
          m_buffer_size(0)
    {
    }

    basic_filemultibuf(basic_filemultibuf const& other)
        : basic_filebuf(other),
          m_buffers(other.m_buffers),
          m_buffer_count(other.m_buffer_count),
          m_curent_buffer(other.m_curent_buffer),
          m_buffer_size(other.m_buffer_size)
    {
    }

    basic_filemultibuf(FILE* f = NULL)
        : basic_filemultibuf(basic_filebuf(f))
    {
    }

    basic_filemultibuf* pubsetbuf(char** buffers, std::ptrdiff_t buffer_count, std::streamsize buffer_size)
    {
        if ((this->m_buffers = buffers) != NULL)
        {
            this->m_buffer_count  = buffer_count;
            this->m_buffer_size   = buffer_size;
            this->m_curent_buffer = 0;
        }
        else
        {
            this->m_buffer_count  = 0;
            this->m_buffer_size   = 0;
            this->m_curent_buffer = -1;
        }

        this->basic_filebuf::setbuf(NULL, 0);
        return this;
    }
};

Example usage:

typedef basic_filemultibuf<char> filemultibuf;

std::fstream file("file", std::ios_base::binary | std::ios_base::in | std::ios_base::out);

char** buffers = new char*[2];
for (int i = 0; i < n; ++i)
    buffers[i] = new char[4096];

filemultibuf multibuf(*file.rdbuf());
multibuf.pubsetbuf(buffers, 2, 4096);
file.set_rdbuf(&multibuf);

//
// do awesome stuff with file ...
//

for (int i = 0; i < n; ++i)
    delete[] buffers[i];

That's pretty much it. The only thing I would really like to do is offer this functionally for other streambufs, because the usage of multiple buffers should not be restricted to filebuf. But it seems to me it isn't possible without rewriting the file specific functions.

What do you think about that?

std::fstream with multiple buffers?

3 Answers3