8

I'm in front of a piece of code, which copies a file to a usb-device. Following part is the important one:

while((bytesRead = fread(buf, 1, 16*1024, m_hSource)) && !bAbort) {
    // write to target
    long bytesWritten = fwrite(buf, 1, bytesRead, m_hTarget);

    m_lBytesCopied += bytesWritten;

The thing, the customer said, it's pretty slow in comparison to normal pc<->usb speed. I didn't code this, so it's my job, to optimize.

So I was wondering, if it's a better approach to first read the complete file and then write the file in one step. But I don't know how error-prone this would be. The code also check after each copystep if all bytes where written correctly, so that might also slow down the process.

I'm not that c++ & hardware guru, so I'm asking you guys, how I could speed things up and keep the copying successful.

Shiplu Mokaddim
  • 56,364
  • 17
  • 141
  • 187
Johannes Klauß
  • 10,676
  • 16
  • 68
  • 122
  • 2
    Look elsewhere for the bottleneck. You may get a small improvement changing the buffer size, but your problem is probably something else. – William Pursell Jan 16 '12 at 11:29
  • Well, that's the only part that is doing something with this particular problem. So it can be only the code. – Johannes Klauß Jan 16 '12 at 11:31
  • Remove the check that the bytes were written correctly. If you cannot trust the first write, then you cannot trust the following read you use for verification. (I assume you are checking bytesWritten for errors.) – William Pursell Jan 16 '12 at 11:35
  • The posted code looks fine when it comes to performance. Is there anything in the part of the code you didn't post that may be causing a problem? – Klas Lindbäck Jan 16 '12 at 11:44
  • Rather than reading back the data to verify it, use fsync(). – William Pursell Jan 16 '12 at 11:46
  • I have heard that sometimes it may help to switch the size and count parameters. Try using `fwrite(buf, bytesRead, 1, m_hTarget)` and see if it helps. Note that this won't work for fread since if you specify size being 1000, and there less than 1000 bytes left to read, the function will fail. – Tom Knapen Jan 16 '12 at 11:50
  • 1
    So, perhaps the device is recognized as a USB 1.1 device. USB 1 is dead slow.. – nos Jan 16 '12 at 11:56
  • Go for copying file using memory mapped I/O...just a suggestion! – cyber_raj Jan 16 '12 at 12:51

3 Answers3

3
  1. Try to read/write in big chunk. 16M, 32M are not bad for copying file.
  2. If you just want to copy the file you can always invoke system() It'll be faster.

  3. The code also check after each copystep if all bytes where written correctly, so that might also slow down the process.

    You can check it by creating hash of bigger chunk. Like splitting the file into 64M chunks. Then match hashes of those chunks. Bittorrent protocol has this feature.

  4. If you have mmap or MapViewOfFile available, map the file first. Then write it to usb. This way read operation will be handled by kernel.

  5. Kerrek just commented about using memcpy on mmap. memcpy with 2 mmaped file seems great.

Also note that, Most recent operating systems writes to USB stick when they are being removed. Before removal it just writes the data in a cache. So copy from OS may appear faster.

Shiplu Mokaddim
  • 56,364
  • 17
  • 141
  • 187
1

What about overlapping reads and writes?

In the current code, the total time is time(read original) + time(write copy), if you read the first block, then while writing it start reading the second block, etc. your total time would be max(time(read original), time(write copy)) (plus the time reading/writing the first and last blocks that won't be pipelined).

It could be almost half the time if reading and writing takes more or less the same time.

You can do it with two threads or with asynchronous IO. Unfortunately, threads and async IO are platform dependent, so you'll have to check your system manual or choose appropriate portable libraries.

fortran
  • 74,053
  • 25
  • 135
  • 175
  • AFAIK, USB I/O is also mostly platform dependent, is it not? – André Caron Jan 16 '12 at 14:56
  • @AndréCaron I guess yes if you are building a driver, but in this case I think it is just accessing the filesystem, so you can `fopen` files (and that is standard). – fortran Jan 16 '12 at 15:03
0

I would just go with some OS specific functions that for sure do this faster that anything written only with c/c++ functions.

For Linux this could be sendfile function. For Windows CopyFile will do the job.

Zuljin
  • 2,612
  • 17
  • 14