Why does this code fail when executed via TPL/Tasks?

Question

I am using System.Net.Http to use network resources. When running on a single thread it works perfectly. When I run the code via TPL, it hangs and never completes until the timeout is hit.

What happens is that all the threads end up waiting on the sendTask.Result line. I am not sure what they are waiting on, but I assume it is something in HttpClient.

The networking code is:

using (var request = new HttpRequestMessage(HttpMethod.Get, "http://google.com/"))
{
    using (var client = new HttpClient())
    {
        var sendTask = client.SendAsync
              (request, HttpCompletionOption.ResponseHeadersRead);
        using (var response = sendTask.Result)
        {
            var streamTask = response.Content.ReadAsStreamAsync();
            using (var stream = streamTask.Result)
            {
                // problem occurs in line above
            }
        }
    }
}

The TPL code that I am using is as follows. The Do method contains exactly the code above.

var taskEnumerables = Enumerable.Range(0, 100);
var tasks = taskEnumerables.Select
            (x => Task.Factory.StartNew(() => _Do(ref count))).ToArray();
Task.WaitAll(tasks);

I have tried a couple of different schedulers, and the only way that I can get it to work is to write a scheduler that limits the number of running tasks to 2 or 3. However, even this fails sometimes.

I would assume that my problem is in HttpClient, but for the life of me I can't see any shared state in my code. Does anyone have any ideas?

Thanks, Erick

I believe winhttp is limited to only two concurrent connection to the same server. It was restricted on XP for sure, later the limit was lifted to 4 concurrent but I don't remember if that was in Vista or 7. On what OS are you? — rene, Oct 22 '12 at 19:16
Why did you pass `HttpCompletionOption.ResponseHeadersRead`? Is the resource you are trying to consume some how a steaming API? http://www.tugberkugurlu.com/archive/streaming-with-newnet-httpclient-and-httpcompletionoption-responseheadersread — tugberk, Oct 22 '12 at 20:02
Rene - the HttpClient can handle multiple calls. I'm not sure of how it works. Tugberk - this is a stripped down example. In the real code, I read the headers to get information needed to use the contents. I've tested, and it has no impact on the problem. — Erick T, Oct 22 '12 at 20:05
Have you tried passing an HttpMessageHandler (WebRequestHandler) to the HttpClient constructor? It might be an authentication or proxy issue. — Oppositional, Oct 22 '12 at 23:31
The code fails on the streamTask.Result line. It blocks there and never continues. It isn't an auth or proxy issue, as the code works perfectly on a single thread. — Erick T, Oct 23 '12 at 17:26
@ErickT Is it possible that you are introducing a deadlock? If your code is as above, it shouldn't but you said that this is a stripped down example. If you are using new asynchronous language features somewhere, it's likely that u will end up with a deadlock if u are not careful enough cuz as I can see, you are making blocking calls. Have a look at here: http://www.tugberkugurlu.com/archive/asynchronousnet-client-libraries-for-your-http-api-and-awareness-of-async-await-s-bad-effects — tugberk, Oct 23 '12 at 18:23

score 2 · Accepted Answer · answered Apr 16 '13 at 19:20

I finally found the issue. The problem was that HttpClient issues its own additional tasks, so a single task that I start might actually end spawning 5 or more tasks.

The scheduler was configured with a limit on the number of tasks. I started the task, which caused the number of running tasks to hit the max limit. The HttpClient then attempted to start its own tasks, but because the limit was reached, it blocked until the number of tasks went down, which of course never happened, as they were waiting for my tasks to finish. Hello deadlock.

The morals of the story:

Tasks might be a global resource
There are often non-obvious interdependencies between tasks
Schedulers are not easy to work with
Don't assume that you control either schedulers or number of tasks

I ended up using another method to throttle the number of connections.

Erick

Why does this code fail when executed via TPL/Tasks?

1 Answers1

Linked