can I gain performance by controlling the buffer flush in std::ofstream?

Question

I am writing my own logger. Hoping to gain some performance, I added a piece of code to control the number of flushes to std::ofstream. To do this, I used a temporary buffer of type std::stringstream. The log operations are written into this buffer first and then flushed into the std::ofstream in the right time. (look at void flushLog()) :

#include<iostream>
#include<sstream>
#include<fstream>
class BasicLogger
{
    std::stringstream out;
    std::ofstream logFile;

    typedef std::basic_ostream<char, std::char_traits<char> > CoutType;
    typedef CoutType& (*StandardEndLine)(CoutType&);
    public:

    BasicLogger(std::string id_){
        std::string path = id_ + ".txt";
        if(path.size()){            
            logFile.open(path.c_str());
            if ((logFile.is_open() && logFile.good())){
            }
        }
    }
    BasicLogger& operator<<(StandardEndLine manip) {
        std::cout << "Blogger:call to cout type oprtor" << std::endl;
        manip(out);
    return *this;
    }

    template <typename T>
    BasicLogger & operator<< (const T& val)
    {
        std::cout << "Blogger:call to oprtor" << std::endl;
        out << val;
        if(out.tellp() > 512000/*500KB*/){// by some googling this estimated hardcode value promises less cycles to write to a file
            flushLog();
        }
        return *this;
    }

void flushLog()
{
    if ((logFile.is_open() && logFile.good()))
    {
            logFile << out.str();
            logFile.flush();
            out.str(std::string());
    }
}
};

coming to know std::ofstream already has its own buffer, I need to rethink if manipulating the buffer was a right thing to do. ??

`out->tellp()`: `out` isn't a pointer. – user657267 Sep 01 '14 at 05:13 — user657267, Sep 01 '14 at 05:13
@user657267 just consider it as typo – rahman Sep 01 '14 at 05:32 — rahman, Sep 01 '14 at 05:32

score 0 · Answer 1 · answered Sep 02 '14 at 00:14

0

You need to consider that a log file shouldn't have too much write latency, as you need to be able to examine recent log entries in case of a failure.
Output buffering has diminishing returns as you increase the buffer size. Going from zero to 1 byte doubles your speed, as it cuts system calls in half; going from 1 to 4096 or 8192 is a major win as it corresponds to a disk block size; going from 8192 to 512k isn't likely to make much difference at all. I suggest you benchmark this before you over-commit yourself.

answered Sep 02 '14 at 00:14

user207421

305,947
44
307
483

the 2nd point is useful, however, I still need to get answer to the mainquestion: should I use my own buffering system as shown above, or tune the buffer size of `ofstream` and leave the rest of the things to it? need someone who can make more sense of posts like http://stackoverflow.com/questions/12997131/stdfstream-buffering-vs-manual-buffering-why-10x-gain-with-manual-buffering and help me out with a suggestion. – rahman Sep 02 '14 at 03:24
EJP are you still following this? I need some more info about your statement ` going from 1 to 4096 or 8192 is a major win as it corresponds to a disk block size` . does operating system and architecture matter? can you send me a reference? thanks – rahman Sep 12 '14 at 08:43

can I gain performance by controlling the buffer flush in std::ofstream?

1 Answers1