.NET HttpClient multithreading

Question

I have some trouble using HttpClient in multiple thread.

If i launch many download simultaneously, the first download of each thread is very slow (and increase with parallel threads)

For example, if I have one thread, everything is fine

First download Elapsed Time Download: 197 ms
Second download Elapsed Time Download: 171 ms

But with one thread, download time increase

First download Elapsed Time Download: 3881 ms
...
Second download Elapsed Time Download: 96 ms
...

Network bandwith is not an issue, i have same problem with localhost.

Here is some code to reproduce problem :

static void Main(string[] args)
{
    ServicePointManager.DefaultConnectionLimit = 200;
    List<Task> tasks = new List<Task>();
    for (var i = 0; i < 10; i++)
    {
        tasks.Add(
            Task.Factory.StartNew(() =>
            {
                Stopwatch s = Stopwatch.StartNew();
                HttpClient httpClient = new HttpClient();
                HttpResponseMessage httpDownloadResponse = httpDownloadResponse = httpClient.GetAsync("http://www.google.fr/", HttpCompletionOption.ResponseHeadersRead).Result;
                s.Stop();
                Console.WriteLine("First download Elapsed Time Download: {0} ms", s.ElapsedMilliseconds);

                s = Stopwatch.StartNew();
                httpClient = new HttpClient();
                httpDownloadResponse = httpClient.GetAsync("http://www.google.fr/", HttpCompletionOption.ResponseHeadersRead).Result;
                s.Stop();
                Console.WriteLine("Second download Elapsed Time Download: {0} ms", s.ElapsedMilliseconds);
            })
        );
    }
    Task.WaitAll(tasks.ToArray());
    while (Console.ReadLine() != null) ;
}

@Adriano - `Console` is threadsafe, and isn't included in the timings. — Brian Reischl, Jul 17 '14 at 13:39
It may be related (or may not) but the convention observed by .net is that only three Http connections to the same URL are allowed from a particular client OS. This is defined in ServicePointManager.DefaultConnectionLimit. This is dicussed here http://stackoverflow.com/questions/866350/how-can-i-programmatically-remove-the-2-connection-limit-in-webclient and may lead to you observing a dramatic slowdown on parallel http activities on the same URL. — PhillipH, Jul 17 '14 at 13:49
@PhillipH - he explicitly set that at the top of the code. `ServicePointManager.DefaultConnectionLimit = 200; ` — Brian Reischl, Jul 17 '14 at 13:52

score 2 · Answer 1 · answered Jul 17 '14 at 14:11

I can't actually reproduce your problem on my machine. No matter what I do, the first request always takes about the same time, and second requests are faster.

My best guess is that you're exhausting the threadpool. Each time you call Task.Factory.StartNew it starts a new Task, which takes a new threadpool thread. Then you call HttpClient.GetAsync, which starts another Task on another thread and blocks the first thread. So you're taking up two threads for each request. If you do that enough you'll use all the threads in the threadpool, and requests will start queuing up. The threadpool will add more threads, but slowly - typically one new thread every 0.5 seconds. So that might be part of it.

Also, you're using HttpClient wrong. Each HttpClient instance holds a connection pool, so you generally want to create one instance and reuse it. Here is some cleaner code - try it and see if it solves your problem.

public static void Main()
{
    ServicePointManager.DefaultConnectionLimit = 200;
    List<Task> tasks = new List<Task>();
    using (var client = new HttpClient())
    {
        for (var i = 0; i < 10; i++)
        {
            tasks.Add(DoRequest(i, client));
        }

        Task.WaitAll(tasks.ToArray());
    }
}

private async Task DoRequest(int id, HttpClient client)
{
    const string url = "http://www.google.fr/";
    Stopwatch s = Stopwatch.StartNew();
    HttpResponseMessage httpDownloadResponse = await client.GetAsync(url, HttpCompletionOption.ResponseHeadersRead);
    s.Stop();
    Console.WriteLine("Task {1} - First download Elapsed Time Download: {0} ms", s.ElapsedMilliseconds, id);

    s = Stopwatch.StartNew();
    httpDownloadResponse = await client.GetAsync(url, HttpCompletionOption.ResponseHeadersRead);
    s.Stop();
    Console.WriteLine("Task {1} - Second download Elapsed Time Download: {0} ms", s.ElapsedMilliseconds, id);

}

Your solution works, but i can't do this. I really need to have multiple threads, which initiate downloads. To be more precise, i will have 10s of long running threads, listening MSMQ queues, which launch download when receiving messages. — Mickael Magniez, Jul 18 '14 at 07:06
Adding ThreadPool.SetMinThreads(100, 100); makes execution time much better. Min threads for correct behaviour is number_tasks + 1 — Mickael Magniez, Jul 18 '14 at 07:15
My code does use threads, it just uses fewer threads more efficiently. If you run it you will see that the are executing in parallel. Under the covers the framework will start each request, and when it finishes one of the I/O Completion threads from the threadpool will service it. This is much more efficient than tying up a thread for each request. I've actually run the tests myself, and using async this way is quite a bit more scalable, and faster and higher loads. — Brian Reischl, Jul 18 '14 at 14:18

.NET HttpClient multithreading

1 Answers1