0

I am writing a file replicator data backup software using the Qt framework and am experiencing a 'flaw' in the data copying.

If I copy data from an SSD to a HDD the SSD is much faster to read data than the HDD is to write data. Because of this the data that is read fills a buffer before it is written to the HDD.

This wouldn't be a problem except that the progress meter for data copy stalls when the buffer gets filled then when it gets emptied it races to get filled again. So the progress meter indicates a bunch of progress, then it stalls, then it indicates progress, then stalls, etc. It looks pretty ugly compared to constant progress which happens when data is written at a constant rate.

Is there a way around this with Qt? I tried using QFile::flush() but it didn't help.

We have FlushFileBuffers from Win32 api but the documentation says it's inefficient when used after every write to a disk drive device when many writes are being performed separately.

We can call the CreateFile function with the FILE_FLAG_NO_BUFFERING and FILE_FLAG_WRITE_THROUGH flags but is there a way to do this with QFile's?

And then we have the option of completely turning off caching for the target disk as here.

What would be the most elegant solution?

Here's the relevant code:

QFile sourceFile;
QFile targetTempFile;
// ...
QByteArray buffer;
for (int count = 0; !(buffer = sourceFile.read(1000000)).isEmpty() && cancel == false; count++)
{
    int readSize = buffer.size();
    targetTempFile.write(buffer);
    //targetTempFile.flush();
    setProgressMeters(readSize, fileSize, jobSize);
    setTimeLabels(startTime.elapsed(),readSize);
}
Kuba hasn't forgotten Monica
  • 95,931
  • 16
  • 151
  • 313
riverofwind
  • 525
  • 4
  • 17
  • @JoshOrenberg The `A` and `W` suffixes on winapi functions have to do with `char` vs. `wchar_t` interfaces. The modern default is to alias `ApiW` to `Api`, i.e. `CopyFileEx` is same as `CopyFileExW`. – Kuba hasn't forgotten Monica May 05 '18 at 17:47
  • What about that CopyFile2 - is it just a CopyFileEx with a different structure for parameters? – riverofwind May 05 '18 at 20:44

1 Answers1

0

For performant file copying on Windows, you should use CopyFileEx, not QFile. CopyFileEx will use lower level APIs that may result in zero-copy copying on modern hardware. See this example of CopyFileEx integration with Qt.

To obtain a native HANDLE from a QFileDevice (i.e. QFile/QSaveFile), and to use ReOpenFile on a QFile, see this answer.

Kuba hasn't forgotten Monica
  • 95,931
  • 16
  • 151
  • 313
  • How would one modify the CopyFileEx example to be thread-safe if run from a worker object moved to a worker thread via a worker->moveToThread(workerThread); call? – riverofwind May 07 '18 at 03:57
  • @JoshOrenberg That question doesn't belong here - ask in the place where you want to see the modifications. But in general, the `Copier` is inherently thread-safe because it uses a worker thread internally anyway - c.f. `QtConcurrent::run`. There's no point to moving `Copier` to any other thread. – Kuba hasn't forgotten Monica May 07 '18 at 04:09