ofstream::operator<<(streambuf) is a slow way to copy a file

Question

I need a cross-platform, no external library, way of copying a file. In my first pass I came up with (error handling omitted):

char buffer[LEN];
ifstream src(srcFile, ios::in | ios::binary);
ofstream dest(destFile, ios::out | ios::binary);

while (!src.eof()) {
  src.read(buffer, LEN);
  dest.write(buffer, src.gcount());
}

This worked nicely and I knew exactly what it was doing.

Then I found a post on stackoverflow (sorry, can't find a link right now) that says I can replace all of the above code with:

dest << src.rdbuf();

Which is nice and compact, but hides a lot about what it's doing. It also turns out to be really slow because the implementation of ofstream::operator<<(streambuf) moves things 1 character at a time (using snetxc()/sputc()).

Is there a way for me to make this method faster? Is there a drawback to my original method?

Update: There's something inefficient about using operator<<(streambuf) on windows. The .read()/.write() loop looks to always perform better than operator<<.

Also, changing the size of the buffer in the code above does not affect the size of the reads and writes to the hard drive. To do that you need to set the buffers using stream.rdbuf()->pubsetbuf().

Answer 1

I wonder if your fstream is unbuffered by default. GCC 4.5.2 by default uses an internal buffer, but I don't think that's required by the standard. Have you tried using pubsetbuf (see below) to set a buffer for your in/out streams.

A quick test on my system, if I set LEN to 0 (and therefore unbuffered), it took 10 seconds to copy a 1 MB file. With a 4k buffer, it completed in less than a second.

#include <iostream>
#include <fstream>

int main() {
  using namespace std;
  const char* srcFile = "test.in";
  const char* destFile = "test.out";

  ifstream src;
  ofstream dest;

  const int LEN=8192;
  char buffer_out[LEN];
  char buffer_in[LEN];
  if (LEN) {
    src.rdbuf()->pubsetbuf(buffer_in, LEN );
    dest.rdbuf()->pubsetbuf(buffer_out, LEN);
  } else {
    src.rdbuf()->pubsetbuf(NULL, 0 );
    dest.rdbuf()->pubsetbuf(NULL, 0);
  }
  src.open(srcFile, ios::in | ios::binary);
  dest.open(destFile, ios::out | ios::binary);
  dest << src.rdbuf();

}

Answer 2

Of course the src.rdbuf method is slower. It's doing reading and writing at the same time. Unless you're copying to a different harddisk or some form of network or attached storage, that's going to be slower than reading a block and then writing a block.

Just because code is compact does not make it faster.

---

Since you can't overload operator<< for the std::filebuf (since it's already overloaded), there isn't much you can do. It's better to just use the method that works reasonably well.

Answer 3

Try using the C stdio API instead, it can often be faster in many implementations (see this thread for some numbers), though not always. For example:

// Error checking omitted for expository purposes
char buffer[LEN];
FILE *src = fopen(srcFile, "rb");
FILE *dest = fopen(destFile, "wb");

int n;
while ((n = fread(buffer, 1, LEN, src)) > 0)
{
    fwrite(buffer, 1, n, dest);
}

fclose(src);
fclose(dest);