C++ Save string line by line to a file as fast as possible

Question

I was trying to write to a file or save the string s.substr (space_pos) in a vector as fast as possible. I tried to write it to a file with ofstream or to output it with cout but it takes a long time. The size of the text file is 130mb.

This is the code:

fstream f(legitfiles.c_str(), fstream::in );
string s;
while(getline(f, s)){
    size_t space_pos = s.rfind(" ") + 1;

    cout << s.substr(space_pos) << endl;
    ofstream results("results.c_str()");
    results << s.substr(space_pos) << endl;
    results.close();

}
cout << s << endl;
f.close();

Is there a way to write or print the string in a faster way?

Answer 1

Uncouple the C++ stream from the C stream:

std::ios_base::sync_with_stdio(false);

Remove the coupling between cin and cout

std::cin.tie(NULL);

Now don't use std::endl needlessly flushes the fstream buffer after every line, flushing is expensive. You should use a newline escape character \n instead and leave the buffer flushing to the stream.

Also don't build an extra string you don't need. Use a character string_view (which prevents copying)

s.substr(space_pos)

//replace with:
std::string_view  view(s);
view.substr(space_pos);

If you don't have a modern compiler just use C-Strings.

s.data() + space_pos

Answer 2

You are duplicating the substring. I suggest creating a temporary:

ofstream results("results.c_str()");
while(getline(f, s)){
    size_t space_pos = s.rfind(" ") + 1;
    const std::string sub_string(s.substr(space_pos));
    cout << sub_string << "\n";
    results << sub_string << "\n";
}
results.close();

You'll need to profile to see if the next code fragment is faster:

while(getline(f, s))
{
    static const char newline[] = "\n";
    size_t space_pos = s.rfind(" ") + 1;
    const std::string sub_string(s.substr(space_pos));
    const size_t length(sub_string.length());
    cout.write(sub_string.c_str(), length);
    cout.write(newline, 1);
    results.write(sub_string.c_str(), length);
    results.write(newline, 1);
}

The idea behind the 2nd fragment is that you are bypassing the formatting process and directly writing the contents of the string to the output stream. You'll need to measure both fragments to see which is faster (start a clock, run an example at least 1E6 iterations, stop the clock. Take average).

If you want to speed up the file writing, remove the writing to std::cout.

Edit 1: multiple threads
You may be able to get some more efficiency out of this by using multiple threads: "Read Thread", "Processing Thread" and "Writing Thread".
The "Read Thread" reads the lines and appends to a buffer. Start this one first. After a delay, the "Processing Thread" performs the substr method on all the strings.
After N about of strings have been processed, the "Writing Thread" starts and writes the substr strings to the file.

This technique uses double buffering. One thread reads and places data into the buffer. When the buffer is full, the Processing Thread should start processing and placing results into a second buffer. When the 2nd buffer is full, the Writing Thread starts and writes the buffer to the results file. There should be at least 2 "read" buffers and 2 "write" buffers. The amount and size of the buffers should be adjusted to get the best performance from your program.

Answer 3

//Edit: Please note that this answer solves a different problem than that stated in the question. It will copy each line skipping everything from the beginning of the line up to and including the first whitespace.

It might be faster to read the first word of a line and throw it away before getline()ing the rest of it instead of using string::find() and std::substr(). Also you should avoid opening and closing the output file on every iteration.

#include <string>
#include <fstream>

int main()
{
    std::ifstream is{ "input" };
    std::ofstream os{ "output" };
    std::string str;
    str.reserve(1024); // change 1024 to your estimated line length.

    while (is.peek() == ' ' || is >> str, std::getline(is, str)) {
        str += '\n';           // save an additional call to operator<<(char)
        os << str.data() + 1;  // +1 ... skip the space
        // os.write(str.data() + 1, str.length() - 1); // might be even faster
    }
}