How do I calculate the opimal size of the bytes to read from a file using QIODevice:read()?

Question

The thing is, I have to read the file and write its data to another file. But the size might be so big (larger than 8 gb) so I read the files by chunks (1 mb), but I think the optimal size of the chunks can be calculated, so how do I do this? what tools should I use? Here's the code

const int BLOCK_SIZE = 1000000;
if(!(fin.open(QIODevice::ReadOnly)) == false &&
   !(fout.open(QIODevice::WriteOnly)) == false) {\

    long long m_file_size = fin.size();
    QDataStream in(&fin);
    QByteArray arrayOfBytes;
    
    int counting = 0;
    int check = 0;
    long int bytesPerSecond = 0;
    timer.start();
    for(size_t i = 0; i <= m_file_size / BLOCK_SIZE; ++i) {
        arrayOfBytes = fin.read(BLOCK_SIZE);
    
        fout.write(arrayOfBytes);
        bytesPerSecond +=BLOCK_SIZE;

        if ((100 * ((timer.elapsed() + 50) / 100)) > 999  && (100 * ((timer.elapsed() + 50) / 100)) % 1000 == 0 && check != (100 * ((timer.elapsed() + 50) / 100)))
        {
            counting++;
            check = (100 * ((timer.elapsed() + 50) / 100));//(10 * ((timer.elapsed() + 5) / 10));
            bytesPerSecond = 0;
        }
    }
    fin.close();
    fout.close();
}
else{
    qDebug()<<"Failed to open";
}

}

The files are big and I need to track the reading speed, so I check how many bytes where written in 1 second — Isidore, Sep 08 '21 at 00:03

score 0 · Answer 1 · answered Sep 08 '21 at 00:15

0

Instead of implementing a file copy, I suggest to use std::filesystem::copy. There is a code sample on that page.

Although I don't know what chunk size will be used, I would hope that the writers of std found suitable value.

And you can time the execution of that call if you want.

answered Sep 08 '21 at 00:15

Vlad Feinstein

10,960
1
12
27

Seems good, but I also need to track the speed of copying – Isidore Sep 08 '21 at 20:24

score 0 · Accepted Answer · answered Sep 08 '21 at 07:41

Try this: https://doc.qt.io/qt-5/qstorageinfo.html#blockSize

If you are interested in tracking copying progress AND optimize the copying for speed at the same time, you might need to write platform specific code. On linux it might be sendfile(). On Windows you may need to call WinAPI... But I would start with some naive solution (like the one you have), then try to optimize buffer size (do measurements to measure the speed gains for each change) and only then try to use platform-native calls (and measure again).

You may also have a look at this answer: Copy a file in a sane, safe and efficient way

And also this: Why is copying a file in C so much faster than C++?

PS: when measuring improvements, be careful about warm-up phase. Copying will be of course much faster if the file is already in the cache... So first copying will be always slower and subsequent copying will be usually faster regardless of the method. In other words, you must make sure to compare "apples-to-apples".

How do I calculate the opimal size of the bytes to read from a file using QIODevice:read()?

2 Answers2