Logging using c++ stl stringstream with real-time performance requirements

Question

I'm attempting to implement logging functionality in an application that is required to run in real-time (meaning, spending a few milliseconds writing to a file during execution of the main loop will significantly affect performance since we are interfacing with several systems that require low latency). My current implementation logs a line into a stringstream object each millisecond, and then writes the resulting string to a file only upon shutdown:

class ControlDemo {
  public:
    ControlDemo();
    ~ControlDemo();
    void spin();
    // Other public methods
  private:
    void logLine(std::vector<double> data);
    void writeLogFile();
    std::stringstream m_logged_data;
    std::string m_log_filename;
    bool m_continue_spinning;
    // Other private methods
};

ControlDemo::~ControlDemo() {
  writeLogFile();  // write log data in RAM to a file
}

void ControlDemo::spin() {
  while(m_continue_spinning){
    // do stuff
    logLine(data_to_log);
    sleepRemainingTime();  // maintaining ~1ms loop rate
  }
}

void ControlDemo::logLine(std::vector<double> data) {
  m_logged_data << getCurrentTime();
  for (std::vector<double>::iterator it = data.begin(); it != data.end(); ++it)
    m_logged_data << ', ' << *it;
  m_logged_data << std::endl;
}

void ControlDemo::writeLogFile() {
  std::ofstream file;
  file.open(m_log_file_path.c_str());

  file << m_logged_data.str();
  file.close();
  std::cerr << "Wrote log to " << m_log_file_path << std::endl;
}

My desire is to implement some type of buffering functionality whereby I can specify the maximum number of lines in my log, after which, the initial lines will be discarded (essentially a circular buffer). In this way, I am not wasting a bunch of RAM and slowing down my application by continuing to accumulate logged data, when maybe only the last 20 seconds of data is important. I attempted this by modifying the logLine() method as follows:

void ControlDemo::logLine(std::vector<double> data) {
  static int line_num = 0;
  if(line_num < m_max_num_lines)
    line_num++;
  else  // discard first line before inserting next line
    m_logged_data.ignore(1000,'\n');  // move get ptr to next line

  m_logged_data << getCurrentTime();
  for (std::vector<double>::iterator it = data.begin(); it != data.end(); ++it)
    m_logged_data << ', ' << *it;
  m_logged_data << std::endl;
}

However the m_logged_data.str() that is called in writeLogFile() still returns all logged data rather than just the last m_max_num_lines, which suggests that moving the stringstream get pointer with ignore() does not affect the size of the buffer used by the stringstream object or optimize the code in any of the ways that I intended.

I feel like either I am missing something obvious about the stringstream class (why isn't there a setBufferSize()?? see this), or I should be using something else to manage the logging functionality. If the latter, any suggestions on what class I should be using? One of the nice things about stringstream is all of the built-in formatting features, which I am using (although I didn't show it in the code snippets above). So, retaining those formatting capabilities would be nice if possible.

Answer 1

If you really like to use std::stringstream, you can create an array of std::stringstream and use it as your own ring buffer. Here is a very small example:

#include <iostream>
#include <fstream>
#include <sstream>

class CyclicStreamer {
public:
    CyclicStreamer(int _n, std::string _filename) : n(_n), cur(0), filename(_filename){
        s = new std::stringstream [n];
    };
    ~CyclicStreamer() {
        delete s;
    };
    void LogLine(int data) {
        s[cur].str(std::string()); // clear the stringstream
        s[cur] << data << std::endl;
        cur =  (cur+1) % n;
    }
    void LogToFile(){
        std::ofstream file;
        file.open(filename.c_str());
        for(int i=cur;i<n;i++){
            file << s[i].str();
        }
        for(int i=0;i<cur;i++){
            file << s[i].str();
        }
        file.close();
    }
private:
    int n;
    int cur;
    std::string filename;
    std::stringstream *s;
};

int main() {
    CyclicStreamer cs = CyclicStreamer(10, "log.txt");
    for(int i=0; i<=20; i++){
        cs.LogLine(i);
    }
    cs.LogToFile();
    return 0;
}

Do note that if you use the example here in your own project with a pretty large m_max_num_lines, you will get into trouble because you might end up using a lot more memories than expected because for each std::stringstream it takes a bit of memory itself. A better way of doing it is to allow each of them to store part of m_max_num_lines. Say, if m_max_num_lines is 1000, then you can have only 10 std::stringstream, each storing a maximum of 100 lines of log. When a std::stringstram has already stored 100 lines of log, cur = (cur+1) % n.

Do note that it would be much better if you can just use a char array and manage it by yourself. You save both memory and time, but as you said, you lose those built-in formatting features， although there're also very nice features of formatting a raw char array, e.g. sprintf().