11

We are building a highly concurrent web application, and recently we have started using asynchronous programming extensively (using TPL and async/await).

We have a distributed environment, in which apps communicate with each other through REST APIs (built on top of ASP.NET Web API). In one specific app, we have a DelegatingHandler that after calling base.SendAsync (i.e., after calculating the response) logs the response to a file. We include the response's basic information in the log (status code, headers and content):

public static string SerializeResponse(HttpResponseMessage response)
{
    var builder = new StringBuilder();
    var content = ReadContentAsString(response.Content);

    builder.AppendFormat("HTTP/{0} {1:d} {1}", response.Version.ToString(2), response.StatusCode);
    builder.AppendLine();
    builder.Append(response.Headers);

    if (!string.IsNullOrWhiteSpace(content))
    {
        builder.Append(response.Content.Headers);

        builder.AppendLine();
        builder.AppendLine(Beautified(content));
    }

    return builder.ToString();
}

private static string ReadContentAsString(HttpContent content)
{
    return content == null ? null : content.ReadAsStringAsync().Result;
}

The problem is this: when the code reaches content.ReadAsStringAsync().Result under heavy server load, the request sometimes hangs on IIS. When it does, it sometimes returns a response -- but hangs on IIS as if it didn't -- or in other times it never returns.

I have also tried reading the content using ReadAsByteArrayAsync and then converting it to String, with no luck.

When I convert the code to use async throughout I get even weirder results:

public static async Task<string> SerializeResponseAsync(HttpResponseMessage response)
{
    var builder = new StringBuilder();
    var content = await ReadContentAsStringAsync(response.Content);

    builder.AppendFormat("HTTP/{0} {1:d} {1}", response.Version.ToString(2), response.StatusCode);
    builder.AppendLine();
    builder.Append(response.Headers);

    if (!string.IsNullOrWhiteSpace(content))
    {
        builder.Append(response.Content.Headers);

        builder.AppendLine();
        builder.AppendLine(Beautified(content));
    }

    return builder.ToString();
}

private static Task<string> ReadContentAsStringAsync(HttpContent content)
{
    return content == null ? Task.FromResult<string>(null) : content.ReadAsStringAsync();
}

Now HttpContext.Current is null after the call to content.ReadAsStringAsync(), and it keeps being null for all the subsequent requests! I know this sounds unbelievable -- and it took me some time and the presence of three coworkers to accept that this was really happening.

Is this some kind of expected behavior? Am I doing something wrong here?

alextercete
  • 4,871
  • 3
  • 22
  • 36
  • 2
    You do realize that calling `ReadContentAsStringAsync` and then immediately calling `Result` on it is basically negating the asynchrony, right? That will block until the job has completed. And `HttpContext.Current` being `null` after the await sounds like it's just not flowing across `await` points, which is irksome but doesn't *entirely* surprise me. You could fetch it at the *start* of your async method and then use that local variable... – Jon Skeet Jul 22 '13 at 21:04
  • I'd probably call wrap a try catch around a call to response.EnsureSuccessStatusCode() before even attempting to process the content. – cgotberg Jul 22 '13 at 21:13
  • 2
    Are you sure that it's always possible to read `HttpResponseMessage.Content`? It seems to me that trying to read your own output stream may not be supported. – Stephen Cleary Jul 22 '13 at 21:37
  • @JonSkeet, yes I am aware of that. The only reason I "blocked" the operation was to try to avoid losing `HttpContext.Current` "forever" -- which I did, but then there was the hanging issue. By the way, I don't think you understood correctly, but `HttpContext.Current` is not being lost until the end of that HTTP Request. It is being lost for subsequent HTTP Requests. I can only get it back with an `iisreset`. – alextercete Jul 23 '13 at 11:45
  • @StephenCleary, what do you mean by that? As far as I am concerned I am not writing a stream directly to `response.Content`, I am using `ObjectContent` instead. – alextercete Jul 23 '13 at 11:49
  • @alextercete: I know of some restrictions where (input) content can only be read once. So are you sure that reading (output) content will work since ASP.NET must read it again after you? – Stephen Cleary Jul 23 '13 at 11:56
  • If you are always using `ObjectContent`, I'd try reading `Value` instead of doing a blocking-asynchronous read of it as a stream. – Stephen Cleary Jul 23 '13 at 11:56
  • @StephenCleary, I am using `ObjectContent` to build a formatted error response (similar to `HttpError`), but when the response is successful I am using the default ASP.NET Web API mechanism (I am not sure which kind of `HttpContent` it uses). I guess I could try to safe cast it to `ObjectContent` and read `Value`, and if it turns out to be null I would fallback to `.ReadAsStringAsync()`. – alextercete Jul 23 '13 at 12:03
  • @alextercete, if you don't want to loose HttpContext.Current, just save it on a "local" variable or pass it as an argument to your method. – Paulo Morgado Jul 24 '13 at 00:43
  • 2017 , I also had this problem (yesterday). It was stuck on `request.Content.ReadAsStringAsync().Result` ( and never released)- I've also asked about it [**here**](http://stackoverflow.com/questions/41423134/100k-sized-post-requests-dont-get-to-webapi). Strange - but setting it to `async` , solved the problem. I still don't know why. [**This is the solution**](https://i.imgur.com/RIxVRcz.jpg)-the left pane was the problematic one while the right pane is the working one. (The left pane **used to work** , but after upgrading to 4.6.2 - it started doing problems) - Again - I don't know why. – Royi Namir Jan 03 '17 at 07:33

2 Answers2

11

I had this problem. Although, I haven't fully tested yet, using CopyToAsync instead of ReadAsStringAsync seems to fix the problem:

var ms = new MemoryStream();
await response.Content.CopyToAsync(ms);
ms.Seek(0, SeekOrigin.Begin);

var sr = new StreamReader(ms);
responseContent = sr.ReadToEnd();
victordscott
  • 381
  • 4
  • 11
  • When I tried this, I was able to read the stream for my security check, but then when WebApi routed to my method, the parameter was null. I got around it by using ReadAsStreamAsync instead and then moving back to the beginning of the stream when I was done. However, with this approach, you have to use the StreamReader constructor that takes 5 arguments and make the last one 'true' to keep the underlying stream open. – esteuart Mar 18 '19 at 21:11
2

With regards to your second issue, the async/await is syntactic sugar for the compiler building a state machine where the call to to a function preceded by "await" returns immediately on the current thread...one that contains HttpContext.Current in its thread local storage. The completion of that async call can occur on a different thread...one that does NOT have HttpContext.Current in its thread local storage.

If you want the completion to execute on the same thread (thus having the same objects in thread local storage like HttpContext.Current), then you need to be aware of this behavior. This is especially important on calls from the main UI thread (if you're building a Windows application) or in ASP.NET, calls from an ASP.NET request thread where you are dependent on HttpContext.Current.

See reference docs on ConfigureAwait(false). Also, view some Channel 9 tutorials on TPL. Once the "easy" stuff is grokked, the presenter will invariably talk about this issue as it causes subtle problems that are not easily understood unless you know what the TPL is doing underneath the covers.

Good luck.

With regards to your first problem, if the caller gets a result, I'm not convinced that IIS has not completed the request. How are you determining that the ASP.NET request thread initiated by this caller is hung in IIS?

triple.vee
  • 126
  • 1
  • 7