Handling Errors in Health Checks

Question

I have some code like this that goes in my Program.cs file:

builder.Services.AddHealthChecks().AddCheck<StartupProbe>("Startup Probe")
    .AddCheck<LivenessProbe>("Liveness Probe")
    .AddCheck<ReadinessProbe>("Readiness Probe");

It adds three IHealthCheck classes to the check the health of my service while running in Kubernetes.

Most of the time, it works just fine. But sometimes it fails with TaskCanceledException stating that "A task was canceled".

These health checks occur every 5 seconds or so, so a cancelled one every now and then is not really a big deal. But I need to suppress the message. They add up till my log files are full of them.

I read about using Exception Handling Middleware, but the docs are confusing me in the following ways:

1. It talks about loading "pages". I am working with a service. It does not have any UI (except for Swagger UI).

2. The Exception Handling Middleware docs seem to want me to decorate a controller with [atrributes] to make it work. But the AddHealthChecks method does not have a controller. It sets up the routes for me.

3. I only want to handle the TaskCanceledException and (preferably) only for the health check calls. But I have not been able to see a way to limit how it handles exceptions.

I also saw the option of using an Exception Filter. But the docs for that say to use Exception Handling Middleware instead of that.

Is there a way to handle errors for my health checks? (And only my health checks)

NOTE: Here is the full exception I get, just in case it is useful:

"EventId": 1,
"LogLevel": "Error",
"Category": "Microsoft.AspNetCore.Diagnostics.DeveloperExceptionPageMiddleware",
"Message": "An unhandled exception has occurred while executing the request.",
"Exception": "
    System.Threading.Tasks.TaskCanceledException: A task was canceled.
    at Microsoft.Extensions.Diagnostics.HealthChecks.DefaultHealthCheckService.CheckHealthAsync(Func\u00602 predicate, CancellationToken cancellationToken)
    at Microsoft.AspNetCore.Diagnostics.HealthChecks.HealthCheckMiddleware.InvokeAsync(HttpContext httpContext)
    at Dynatrace.OneAgent.Introspection.Shared.OwinMiddlewareBase\u00601.Invoke(Context context)
    at Swashbuckle.AspNetCore.SwaggerUI.SwaggerUIMiddleware.Invoke(HttpContext httpContext)
    at Swashbuckle.AspNetCore.Swagger.SwaggerMiddleware.Invoke(HttpContext httpContext, ISwaggerProvider swaggerProvider)
    at Microsoft.AspNetCore.HeaderPropagation.HeaderPropagationMiddleware.Invoke(HttpContext context)
    at Dynatrace.OneAgent.Introspection.Shared.OwinMiddlewareBase\u00601.Invoke(Context context)
    at Microsoft.AspNetCore.Diagnostics.DeveloperExceptionPageMiddleware.Invoke(HttpContext context)",
"State": {
    "Message": "An unhandled exception has occurred while executing the request.",
    "{OriginalFormat}": "An unhandled exception has occurred while executing the request."
}

Answer 1

I got this working. I was able to reproduce this error using a console application (see here for details).

I added a middleware to address the issue. I added the following class:

/// <summary>
/// This middleware will catch any cancellation exceptions (OperationCanceledException or TaskCanceledException) in health checks, and return 204 No Content.
/// Adapted from: https://github.com/dotnet/aspnetcore/issues/28568#issuecomment-784013537
/// Relevant Link: https://stackoverflow.com/questions/73791547/get-httpclient-to-abort-the-connection
/// </summary>
public class HealthCheckCancellationSuppressionMiddleware
{
    private readonly RequestDelegate next;

    public HealthCheckCancellationSuppressionMiddleware(RequestDelegate next)
    {
        this.next = next;
    }

    public async Task InvokeAsync(HttpContext httpContext)
    {
        try
        {
            // Forward to next middleware
            await next(httpContext);
        }
        //Note TaskCanceledException inherits from OperationCanceledException (so this works for both)
        catch (OperationCanceledException)
        {
            if (httpContext != null && httpContext.Request.Path.StartsWithSegments("/health"))
            {
                // If we have a response already, then just let that happen.
                if (httpContext.Response.HasStarted) return;

                // Set a status code. Response will likely not be seen by client as we expect all cancellations to come from httpContext.RequestAborted
                // Why 204 No Content? I could not find any other appropriate status code.
                httpContext.Response.StatusCode = StatusCodes.Status204NoContent;
            }
            else
            {
                throw;
            }
        }
    }
    
}

I then added the above middleware BEFORE the call to AddHealthChecks, like this:

app.UseMiddleware<HealthCheckCancellationSuppressionMiddleware>();

This got rid of the logs of TaskCanceledExceptions.