.net File.Copy very slow when copying many small files (not over network)

Question

I'm making a simple folder sync backup tool for myself and ran into quite a roadblock using File.Copy. Doing tests copying a folder of ~44,000 small files (Windows mail folders) to another drive in my system, I found that using File.Copy was over 3x slower than using a command line and running xcopy to copy the same files/folders. My C# version takes over 16+ minutes to copy the files, whereas xcopy takes only 5 minutes. I've tried searching for help on this topic, but all I find is people complaining about slow file copying of large files over a network. This is neither a large file problem nor a network copying problem.

I found an interesting article about a better File.Copy replacement, but the code as posted has some errors which causes problems with the stack and I am nowhere near knowledgeable enough to fix the problems in his code.

Are there any common or easy ways to replace File.Copy with something more speedy?

Answer 1

One thing to consider is whether your copy has a user interface that updates during the copy. If so, make sure your copy is running on a separate thread, or both your UI will freeze up during the copy, and the copy will be slowed down by making blocking calls to update the UI.

I have written a similar program and in my experience, my code ran faster than a windows explorer copy (not sure about xcopy from the command prompt).

Also if you have a UI, don't update on every file; instead update every X megabytes or every Y files (whichever comes first), this keeps down the amount of updating to something the UI can actually handle. I used every .5MB or 10 files; those may not be optimal but it noticeably increased my copy speed and UI responsiveness.

Another way to speed things up is to use the Enumerate functions instead of Get functions (e.g. EnumerateFiles instead of GetFiles). These functions start returning results as soon as possible instead of waiting to return everything when the list is finished being built. They return an Enumerable, so you can just call foreach on the result: foreach(string file in System.IO.Directory.EnumerateDirectories(path)). For my program this also made a noticeable difference in speed, and would be even more helpful in cases like yours where you are dealing with directories containing many files.

Answer 2

One of the things that slows down IO operations the most on rotational disks is moving the disk head.

It's reasonable to assume and probably quite accurate that your many small files (that all are related to each other) are closer together on the disk than they are close to the destination of the copy (assuming you're copying from one part of a disk to another part of the same disk). If you copy for a bit then write for a bit, you open a window of opportunity for other processes to move the disk head on the source or target disk.

One thing that XCopy does much better than Copy (meaning in both cases the commands) is that XCopy reads in a bunch of files before starting to write out those files to the destination.

If you are copying files on the same disk, try allocating a large buffer to read in many files at once, then write out those files once the buffer is full).

If you are reading from one disk and writing to another disk, try starting up one thread to read from the source disk and a separate thread to write to the other disk.

Answer 3

There are two algorithms for faster file copy:

If source and destination are different disks Then:

• One thread reading files continuously and storing in a buffer.
• Another thread writing files continuously from that buffer.

If source and destination is same disk then:

• Read a fixed chunk of bytes, say 8K at a time, no matter how many files that is.
• Write that fixed chunk to destination, either in one file or in multiple files.

This way you will get significant performance.

Alternative is you just invoke xcopy from your .net code. Why bother doing it using File.Copy. You can capture xcopy output using Process.StandardOutput and show on the screen in order to show user what's going on.

Answer 4

I think you could at least parallize it so that you do two files at the same time. While one thread is writing another can already be reading the next file. If you have a list of the files you can do that like this. Using many threads will not help because this will make the drive move around a lot more instead of being able to write sequentially..

 var files = new List<string>();
 // todo: fill the files list using directoryenumeration or so...
 var po = new ParallelOptions() {MaxDegreeOfParallelism = 2};
 Parallel.ForEach(files, po, CopyAFile);

 // Routine to copy a single file
 private void CopyAFile(string file) { }

Answer 5

I have no good experience at this level. Why don't you try to run a batch file containing your xcopy comand? Check this post: Executing Batch File in C#