DownloadFileTaskAsync downloads only 2 files at a time

Question

I have implemented the code from the below thread, however I see that all the files starts to download but only 2 files downloads at a time

Asynchronously and parallelly downloading files

private static async Task<Tuple<string, string, Exception>> DownloadFileTaskAsync(string remotePath, string localPath = null, int timeOut = 3000)
        {
            try
            {

                localPath = localPath + remotePath.Split('/')[5].ToString();

                using (var client = new WebClient())
                {
                    client.Credentials = new NetworkCredential("user", "password");

                    using (var timer = new Timer(timerCallback, client, timeOut, Timeout.Infinite))
                    {
                        await client.DownloadFileTaskAsync(remotePath, localPath);
                    }
                    Debug.WriteLine(string.Format("DownloadFileTaskAsync (downloaded): {0}", remotePath));
                    return new Tuple<string, string, Exception>(remotePath, localPath, null);
                }
            }
            catch (Exception ex)
            {
                return new Tuple<string, string, Exception>(remotePath, null, ex);
            }
        }


        private async void Button_Click(object sender, RoutedEventArgs e)
        {
            IEnumerable<string> enumerable = new string[] {"https://somepath.com/file1.zip" };
            var results = new List<Tuple<string, string, Exception>>();
            await enumerable.ForEachAsync(s => DownloadFileTaskAsync(s, "C:\\Download\\", 10000), (url, t) => results.Add(t));

        }

    }

    public static class Extensions
    {
        public static Task ForEachAsync<TSource, TResult>(this IEnumerable<TSource> source, Func<TSource, Task<TResult>> taskSelector, Action<TSource, TResult> resultProcessor)
        {
            var oneAtATime = new SemaphoreSlim(5, 10);
            return Task.WhenAll(
                from item in source
                select ProcessAsync(item, taskSelector, resultProcessor, oneAtATime));
        }

        private static async Task ProcessAsync<TSource, TResult>(TSource item, Func<TSource, Task<TResult>> taskSelector, Action<TSource, TResult> resultProcessor, SemaphoreSlim oneAtATime)
        {
            TResult result = await taskSelector(item);
            await oneAtATime.WaitAsync();
            try
            {
                resultProcessor(item, result);
            }
            finally
            {
                oneAtATime.Release();
            }
        }
    }

Webclient by default can only create two concurrent connection at a time, see this for detail: https://social.msdn.microsoft.com/Forums/vstudio/en-US/3c43e247-158c-4b5c-a6fc-c205d258835d/webclient-maximum-simultaneous-downloads?forum=vbgeneral — Deepak Bhatia, May 18 '16 at 05:05

score 2 · Accepted Answer · answered May 18 '16 at 05:00

2

This is because 2 is the default number of concurrent connections allowed by ServicePointManager.

The solution is simple:

System.Net.ServicePointManager.DefaultConnectionLimit = 10; // Whatever you want it to be

answered May 18 '16 at 05:00

Kirill Shlenskiy

9,367
27
39

Thanks much Kirill!. if it increase the number, will it create any performance hit or is there a recommended number – SSK May 18 '16 at 05:17
@SSK, this is scenario-dependent but a quick search reveals that the number `12 * ` keeps coming up. Whether or not this will create any issues is largely dependent on the number of concurrent connections that your *target* hosts are prepared to accept from your IP. You'll find this one out in their API specs or through trial and error. – Kirill Shlenskiy May 18 '16 at 05:59

DownloadFileTaskAsync downloads only 2 files at a time

1 Answers1