C# File.Copy to multiple Network destination block other threads

Question

Hi I am using multithread to copy many files from a source to multi-network destinations, each thread copy a bulk of files to Different network! I use .net File.Copy(...) I see 100% uses on only one network, each moment

the 100% change from network to network.

1. I tried to change the destinations to local one, then i see balanced bytes copy over all threads

2. I tried to run 10 processes(each one to a different destination) instead of 10 thread, then I get all 10 network at 100% use.

I use .net 4.5 any idea ?

Answer 1

I'd suggest you to replace threads (good for CPU bound operations) with async/await model which perform excellent for IO bound operations.

Let's rewrite File.Copy operation as

public static async Task Copy(string src, string dest)
{
    await Task.Run(() => {
        System.IO.File.Copy(src, dest);
    });
}

You can call it from calling method as snippet below.

var srcPath = "your source location";
var dstPath = "your destination location";

foreach(var file in System.IO.Directory.EnumerateFiles(srcPath))
{
    var dstFile = System.IO.Path.Combine(dstPath, System.IO.Path.GetFileName (file));
    Copy (file, dstFile);
}

Now you can simply pump this method with src and destination paths as fast as you can. Your limitation will be IO speed (disk/network etc) but your Cpu will be mostly free.

Answer 2

Have you taken at look at asynchronous operations. There is some really good documentations on MSDN about specifically async IO.

There are also some questions about async IO on stack overflow such as this one.

Using Task.Run you can queue all of your file copy operations asynchronously.

Answer 3

Try something like:

    List<String> fileList = new List<string>();
    fileList.Add("TextFile1.txt");
    fileList.Add("TextFile2.txt");

    Parallel.For(0, fileList.Count, x =>
    {
        File.Copy(fileList[x], @"C:\" + fileList[x]);
    }
    );

Change c:\ to match your multiple destinations. If you provide the orignal code we could do more.