ASP.NET Web API OperationCanceledException when browser cancels the request

Question

When a user loads a page, it makes one or more ajax requests, which hit ASP.NET Web API 2 controllers. If the user navigates to another page, before these ajax requests complete, the requests are canceled by the browser. Our ELMAH HttpModule then logs two errors for each canceled request:

Error 1:

System.Threading.Tasks.TaskCanceledException: A task was canceled.
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter`1.GetResult()
   at System.Web.Http.Controllers.ApiControllerActionInvoker.<InvokeActionAsyncCore>d__0.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter`1.GetResult()
   at System.Web.Http.Controllers.ActionFilterResult.<ExecuteAsync>d__2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Web.Http.Filters.AuthorizationFilterAttribute.<ExecuteAuthorizationFilterAsyncCore>d__2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter`1.GetResult()
   at System.Web.Http.Controllers.ExceptionFilterResult.<ExecuteAsync>d__0.MoveNext()

Error 2:

System.OperationCanceledException: The operation was canceled.
   at System.Threading.CancellationToken.ThrowIfCancellationRequested()
   at System.Web.Http.WebHost.HttpControllerHandler.<WriteBufferedResponseContentAsync>d__1b.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Web.Http.WebHost.HttpControllerHandler.<CopyResponseAsync>d__7.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Web.Http.WebHost.HttpControllerHandler.<ProcessRequestAsyncCore>d__0.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Web.TaskAsyncHelper.EndTask(IAsyncResult ar)
   at System.Web.HttpApplication.CallHandlerExecutionStep.System.Web.HttpApplication.IExecutionStep.Execute()
   at System.Web.HttpApplication.ExecuteStep(IExecutionStep step, Boolean& completedSynchronously)

Looking at the stacktrace, I see that the exception is being thrown from here: https://github.com/ASP-NET-MVC/aspnetwebstack/blob/master/src/System.Web.Http.WebHost/HttpControllerHandler.cs#L413

My question is: How can I handle and ignore these exceptions?

It appears to be outside of user code...

Notes:

• I am using ASP.NET Web API 2
• The Web API endpoints are a mix of async and non-async methods.
• No matter where I add error logging, I am unable to catch the exception in user code
  • Global.asax Applicaiton_Error
  • TaskScheduler.UnobservedTaskException
  • ELMAH Error Filtering void ErrorLog_Filtering (https://code.google.com/p/elmah/wiki/ErrorFiltering)

Answer 1

This is a bug in ASP.NET Web API 2 and unfortunately, I don't think there's a workaround that will always succeed. We filed a bug to fix it on our side.

Ultimately, the problem is that we return a cancelled task to ASP.NET in this case, and ASP.NET treats a cancelled task like an unhandled exception (it logs the problem in the Application event log).

In the meantime, you could try something like the code below. It adds a top-level message handler that removes the content when the cancellation token fires. If the response has no content, the bug shouldn't be triggered. There's still a small possibility it could happen, because the client could disconnect right after the message handler checks the cancellation token but before the higher-level Web API code does the same check. But I think it will help in most cases.

David

config.MessageHandlers.Add(new CancelledTaskBugWorkaroundMessageHandler());

class CancelledTaskBugWorkaroundMessageHandler : DelegatingHandler
{
    protected override async Task<HttpResponseMessage> SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
    {
        HttpResponseMessage response = await base.SendAsync(request, cancellationToken);

        // Try to suppress response content when the cancellation token has fired; ASP.NET will log to the Application event log if there's content in this case.
        if (cancellationToken.IsCancellationRequested)
        {
            return new HttpResponseMessage(HttpStatusCode.InternalServerError);
        }

        return response;
    }
}

Answer 2

When implementing an exception logger for WebApi, it is recommend to extend the System.Web.Http.ExceptionHandling.ExceptionLogger class rather than creating an ExceptionFilter. The WebApi internals will not call the Log method of ExceptionLoggers for canceled requests (however, exception filters will get them). This is by design.

HttpConfiguration.Services.Add(typeof(IExceptionLogger), myWebApiExceptionLogger); 

Answer 3

Here's an other workaround for this issue. Just add a custom OWIN middleware at the beginning of the OWIN pipeline that catches the OperationCanceledException:

#if !DEBUG
app.Use(async (ctx, next) =>
{
    try
    {
        await next();
    }
    catch (OperationCanceledException)
    {
    }
});
#endif

Answer 4

I have found a bit more details on this error. There are 2 possible exceptions that can happen:

1. OperationCanceledException
2. TaskCanceledException

The first one happens if connection is dropped while your code in controller executes (or possibly some system code around that as well). While the second one happens if the connection is dropped while the execution is inside an attribute (e.g. AuthorizeAttribute).

So the provided workaround helps to mitigate partially the first exception, it does nothing to help with the second. In the latter case the TaskCanceledException occurs during base.SendAsync call itself rather that cancellation token being set to true.

I can see two ways of solving these:

1. Just ignoring both exceptions in global.asax. Then comes the question if it's possible to suddenly ignore something important instead?
2. Doing an additional try/catch in the handler (though it's not bulletproof + there is still possibility that TaskCanceledException that we ignore will be a one we want to log.

---

config.MessageHandlers.Add(new CancelledTaskBugWorkaroundMessageHandler());

class CancelledTaskBugWorkaroundMessageHandler : DelegatingHandler
{
    protected override async Task<HttpResponseMessage> SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
    {
        try
        {
            HttpResponseMessage response = await base.SendAsync(request, cancellationToken);

            // Try to suppress response content when the cancellation token has fired; ASP.NET will log to the Application event log if there's content in this case.
            if (cancellationToken.IsCancellationRequested)
            {
                return new HttpResponseMessage(HttpStatusCode.InternalServerError);
            }
        }
        catch (TaskCancellationException)
        {
            // Ignore
        }

        return response;
    }
}

The only way I figured out we can try to pinpoint the wrong exceptions is by checking if stacktrace contains some Asp.Net stuff. Does not seem very robust though.

P.S. This is how I filter these errors out:

private static bool IsAspNetBugException(Exception exception)
{
    return
        (exception is TaskCanceledException || exception is OperationCanceledException) 
        &&
        exception.StackTrace.Contains("System.Web.HttpApplication.ExecuteStep");
}

Answer 5

You could try changing the default TPL task exception handling behavior through web.config:

<configuration> 
    <runtime> 
        <ThrowUnobservedTaskExceptions enabled="true"/> 
    </runtime> 
</configuration>

Then have a static class (with a static constructor) in your web app, which would handle AppDomain.UnhandledException.

However, it appears that this exception is actually getting handled somewhere inside ASP.NET Web API runtime, before you even have a chance to handle it with your code.

In this case, you should be able to catch it as a 1st chance exception, with AppDomain.CurrentDomain.FirstChanceException, here is how. I understand this may not be what you are looking for.

Answer 6

I sometimes get the same 2 exceptions in my Web API 2 application, however i can catch them with the Application_Error method in Global.asax.cs and using a generic exception filter.

The funny thing is, though, i prefer not to catch these exceptions, because i always log all the unhandled exceptions that can crash the application (these 2, however, are irrelevant for me and apparently don't or at least shouldn't crash it, but i may be wrong). I suspect these errors show up due to some timeout expiration or explicit cancellation from the client, but i would have expected them to be treated inside the ASP.NET framework and not propagated outside of it as unhandled exceptions.

Answer 7

The OP mentioned the desire to ignore System.OperationCanceledException within ELMAH and even provides a link in the right direction. ELMAH has come a long way since the original post and it provides a rich capability to do exactly what the OP is requesting. Please see this page (still being completed) which outlines the programmatic and declarative (configuration-based) approaches.

My personal favorite is the declarative approach made directly in the Web.config. Follow the guide linked above to learn how to set up your Web.config for configuration-based ELMAH exception filtering. To specifically filter-out System.OperationCanceledException, you would use the is-type assertion as such:

<configuration>
  ...
  <elmah>
  ...
    <errorFilter>
      <test>
        <or>
          ...
          <is-type binding="BaseException" type="System.OperationCanceledException" />
          ...
        </or>
      </test>
    </errorFilter>
  </elmah>
  ...
</configuration>

Answer 8

We have been receiving the same exception, we tried to use @dmatson's workaround, but we'd still get some exception through. We dealt with it until recently. We noticed some Windows logs growing at an alarming rate.

Error files located in: C:\Windows\System32\LogFiles\HTTPERR

Most of the errors's were all for "Timer_ConnectionIdle". I searched around and it seemed that even though the web api call was finished, the connection still persisted for two minutes past the original connection.

I then figured we should try to close the connection in the response and see what happens.

I added response.Headers.ConnectionClose = true; to the SendAsync MessageHandler and from what I can tell the clients are closing the connections and we aren't experiencing the issue any more.

I know this isn't the best solution, but it works in our case. I'm also i'm pretty sure performance-wise this isn't something you'd want to do if your API is getting multiple calls from the same client back-to-back.

Answer 9

In case you are using sentry SDK you can ignore it in the options of UseSentry() like this:

public class Program
{
    public static void Main(string[] args)
    {
        CreateHostBuilder(args).Build().Run();
    }

    public static IHostBuilder CreateHostBuilder(string[] args) =>
        Host.CreateDefaultBuilder(args)
            .ConfigureWebHostDefaults(webBuilder =>
            {
                webBuilder
                    .UseSentry(options =>
                    {
                        options.AddExceptionFilterForType<OperationCanceledException>();
                    })
                    .UseContentRoot(Directory.GetCurrentDirectory())
                    .UseStartup<Startup>();
            });
}

Link to the docs: https://sentry-docs-git-sentry-ruby-40.sentry.dev/platforms/dotnet/guides/aspnetcore/ignoring-exceptions

According to the docs:

SentrySdk.Init(o => o.AddExceptionFilterForType<OperationCanceledException>());

Should also work with newer versions of ASP.NET WebApi.

Answer 10

This solution is derived from a solution provided above by @dmatson and other supporting members and with this code I was able to resolve my issue.

The Code:

//add handler to config
    config.MessageHandlers.Add(new CancelledTaskBugWorkaroundMessageHandler());
 
    // create handler
    class CancelledTaskBugWorkaroundMessageHandler : DelegatingHandler
    {
        protected override async Task<HttpResponseMessage> SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
        {
            try
            {
                HttpResponseMessage response = await base.SendAsync(request, cancellationToken);
                // Try to suppress response content when the cancellation token has fired; ASP.NET will log to the Application event log if there's content in this case.
                if (cancellationToken.IsCancellationRequested)
                {
                    return new HttpResponseMessage(HttpStatusCode.OK);
                }
                return response;

            }
            catch (Exception ex)
            {
                if(IsAspNetBugException(ex))
                    return new HttpResponseMessage(HttpStatusCode.OK);
                return new HttpResponseMessage(HttpStatusCode.InternalServerError);
            }


        }
        private static bool IsAspNetBugException(Exception exception)
        {
            return
                (exception is TaskCanceledException || exception is OperationCanceledException)
                &&
                exception.StackTrace.Contains("System.Web.HttpApplication.ExecuteStep");
        }

Note: You can return response message of your choice i.e Internal server or OK personally, I prefer OK if the issue is not from code, however if it's just an operation that was cancelled from an user and we can't do anything about it other than just log the error.