As the title suggests, I have this highly unusual situation that I just can't explain so as a last resort I am posting it here.
It's best I illustrate it with code right away.
I have the following situation somewhere in my IHttpHandler implementation:
var requestData = /*prepare some request data for result service...*/;
//Call the service and block while waiting for the result
MemoryStream result = libraryClient.FetchResultFromService(requestData).Result;
//write the data back in the form of the arraybuffer
HttpContext.Current.Response.BinaryWrite(CreateBinaryResponse(result));
FetchResultFromService is a asynchronous library method that uses HttpClient to retrieve the result from a remote service. Simplified version looks like this:
public async Task<MemoryStream> FetchResultFromService<T>(T request)
{
...//resolve url, formatter and media type
//fire the request using HttpClient
HttpResponseMessage response = await this.client.PostAsync(
url,
request,
formatter,
mediaType);
if (!response.IsSuccessStatusCode)
{
throw new Exception("Exception occurred!");
}
//return as memory stream
var result = new MemoryStream();
await response.Content.CopyToAsync(result);
result.Position = 0;
return result;
}
Issue happens when FetchResultFromService takes more than 2 mins to respond. If that occurs, IIS aborts the blocked thread(IIS timeout is set to 120sec) with an ThreadAbortException. Client(browser) gets the error response and everything seems okay for the moment, but short while after that performance of the app drops and response times skyrocket.
Looking at the logs of the "Library service" it's clear that the moment when service is finally done(2+ minutes) and returns the response is the moment when the anomaly on the webserver occurs. The issue usually resolves itself a minute or two after it starts.
Hopefully, the title is now clearer. When there is a continuation of the task being executed AFTER the caller thread finished the request whole application suffers.
If I modify the library code with ConfigureAwait(false) this issue doesn't occur.
HttpResponseMessage response = await this.client.PostAsync(
url,
request,
formatter,
mediaType).ConfigureAwait(false);//this fixes the issue
So now it looks like it's related to context synchronization. The application is using the old LegacyAspNetSynchronizationContext which does locking over HttpAplication objects but I can't understand how this can affect all threads/requests.
Now I understand that there are a lot of bad practices illustrated above. I am aware of articles like Don't block on Async Code and while I am grateful for all responses I get, I am more looking for an explanation of how can one single async request bring an entire application to its knees in this situation.
More information
- We are not talking about deadlocks because of the blocking. The above situation executes without an issue if the service manages to return the result in those 2 minutes that IIS allows it to do so. The issue can even be reproduced if you don't block and just invoke the library method without calling the
Resultgetter . - Removing second await(the
MemoryStreamcopying) reduces reproduction rate by a lot, but it's still achievable with multiple LibraryFetchResultFromServicecalls fired at the same time.ConfigureAwait(false)is currently the only reliable "solution". - I've even caught IIS even apparently dropping ACTIVE connections. I've been using JMeter to stress test the application and when the anomaly happens JMeter threads(fake users) report that connection was reset. While manually testing with Chrome I've noticed that drop as well with
ERR_CONNECTION_RESETresponse. These resets happen regularly when the application is under load and anomaly occurs, but as mentioned above anomaly always manifests like a performance hit for most of the active users. - The Application is a WebForms app which implements and uses a custom synchronous
IHttpHandlerfor most of its requests. The Handler implementation is declared as IRequiresSessionState promoting exclusive session access. I am mentioning this since I have a hunch that this might be related with session states although I can't confirm it. Or maybe it's related to calling async methods from sync handlers, although still would like to know how. - IIS version is 7 and IIS logs don't show anything useful(or at all).
I am out of ideas for testing, hopefully someone can at least hint me in some direction. I would really love to understand this. Thanks!
