C# WebClient with Task.Run only achieve 5% network usage. WHY?

Question

I am experimenting / learning the new Task library and I have written a very simple html downloader using WebClient and Task.Run. However I can never reach anything more than 5% on my network usage. I would like to understand why and how I can improve my code to reach 100% network usage / throughput (probably not possible but it has to be a lot more than 5%).

I would also like to be able to limit the number of thread however it seems it's not as easy as I thought (i.e. custom task scheduler). Is there a way to just do something like this to set the max thread count: something.SetMaxThread(2)?

internal static class Program
    {
        private static void Main()
        {
            for (var i = 0; i < 1000000; i++)
            {
                Go(i, Thread.CurrentThread.ManagedThreadId);
            }

            Console.Read();
        }

        private static readonly Action<int, int> Go = (counter, threadId) => Task.Run(() =>
        {
            var stopwatch = new Stopwatch();
            stopwatch.Start();

            var webClient = new WebClient();
            webClient.DownloadString(new Uri("http://stackoverflow.com"));

            stopwatch.Stop();

            Console.Write("{0} == {1} | ", threadId.ToString("D3"), Thread.CurrentThread.ManagedThreadId.ToString("D3"));
            Console.WriteLine("{0}: {1}ms ", counter.ToString("D3"), stopwatch.ElapsedMilliseconds.ToString("D4"));
        });
    }

This is the async version according to @spender. However my understanding is that await will "remember" the point in time and hand off the download to OS level and skip (the 2 console.write) and return to main immediately and continue scheduling the remaining Go method in the for loop. Am I understanding it correctly? So there's no blocking on the UI.

private static async void Go(int counter, int threadId)
{
    using (var webClient = new WebClient())
    {
        var stopWatch = new Stopwatch();
        stopWatch.Start();

        await webClient.DownloadStringTaskAsync(new Uri("http://ftp.iinet.net.au/test500MB.dat"));

        stopWatch.Stop();

        Console.Write("{0} == {1} | ", threadId.ToString("D3"), Thread.CurrentThread.ManagedThreadId.ToString("D3"));
        Console.WriteLine("{0}: {1}ms ", counter.ToString("D3"), stopWatch.ElapsedMilliseconds.ToString("D4"));
    }
}

What I noticed was that when I am downloading large files there's no that much difference in terms of download speed / network usage. They (threading version and the async version) both peaked at about 12.5% network usage and about 12MByte download /sec. I also tried to run multiple instances (multiple .exe running) and again there's no huge difference between the two. And when I am trying to download large files from 2 URLs concurrently (20 instances) I get similar network usage (12.5%) and download speed (10-12MByte /sec). I guess I am reaching the peak?

Answer 1

As it stands, your code is suboptimal because, although you are using Task.Run to create asynchronous code that runs in the ThreadPool, the code that is being run in the ThreadPool is still blocking on the line:

webClient.DownloadString(...

This amounts to an abuse of the ThreadPool because it is not designed to run blocking tasks, and is slow to spin up additional threads to deal with peaks in workload. This in turn will have a seriously degrading effect on the smooth running of any API that uses the ThreadPool (timers, async callbacks, they're everywhere), because they'll schedule work that goes to the back of the (saturated) queue for the ThreadPool (which is tied up reluctantly and slowly spinning up hundreds of threads that are going to spend 99.9% of their time doing nothing).

Stop blocking the ThreadPool and switch to proper async methods that do not block.

So now you can literally break your router and seriously upset the SO site admins with the following simple mod:

   private static void Main()
    {
        for (var i = 0; i < 1000000; i++)
        {
            Go(i, Thread.CurrentThread.ManagedThreadId);
        }

        Console.Read();
    }

    private static async Task Go(int counter, int threadId)
    {
        var stopwatch = new Stopwatch();
        stopwatch.Start();

        using (var webClient = new WebClient())
        {
            await webClient.DownloadStringTaskAsync(
                             new Uri("http://stackoverflow.com"));
        }
            //...
    }

HttpWebRequest (and therefore WebClient) are also constrained by a number of limits.