Asp.Net Core HealthChecks Concurrent Entity Framework DbContext Errors

Question

I read this: https://github.com/dotnet/aspnetcore/issues/14453 and it seems i have the same problem.

I have around 50 projects in my solution and each has their own RegisterServices.cs which defines the db context

 services.AddDbContextPool<Db.ACME.Context>(
          options => options.UseSqlServer(configuration.GetConnectionString("ACME"))); 

In there I also add the healthchecks but as soon as I add more than 1 e.g. in the project "PROJECTS" one for checking duplicate projectnames and one for checking for checking duplicate task names.

services.AddHealthChecks()
           .AddCheck<DuplicateTaskNamesHealthCheck>("Duplicate Task Names", HealthStatus.Unhealthy,
               tags: new[] { "org_project" });

        services.AddHealthChecks()
           .AddCheck<DuplicateProjectNamesHealthCheck>("Duplicate Project Names", HealthStatus.Unhealthy,
               tags: new[] { "org_project" });

Where the Healthcheck simply calls the service "projectservice" and the other "taskservice" (both in the same vs solution project)

It will fail with the known

A second operation was started on this context before a previous operation completed. This is usually caused by different threads concurrently using the same instance of DbContext. For more information on how to avoid threading issues with DbContext, see https://go.microsoft.com/fwlink/?linkid=2097913.

The Healthchecks themselves do not contain any logic but simply call the specific service where the logic (EF linq queries) resides. When I comment either one out, they will work.

It seems the only solution is to copy functionality from the services inside the healthchecks but the disturbing thing is that this is not really handy. The more healthchecks, the more methods will be copied over and DRY is out of the window.

Is there any other known workaround?

---

example of a method called in the project service that fails

public async Task<List<string>> GetDuplicateProjectNamesAsync()
    {
        IQueryable<IGrouping<string, PROJECT>> DuplicateProjectNames = _context
            .PROJECT
            .AsNoTracking()
            .GroupBy(x => x.PROJECTNAME)
            .Where(x => x.Count() > 1);
        
        /* next line fails */
        var hasItems = await DuplicateProjectNames.AnyAsync();

        if (hasItems)
        {
            return await DuplicateProjectNames.Select(x=>x.Key).ToListAsync();
        }
        else 
        { 
            return new List<string>();
        }
    }

---

example of the healthcheck

 namespace ACME.Org.Project.Healthchecks
 { 
public class DuplicateProjectNamesHealthCheck : IHealthCheck
{
    public IACME6_PROJECT _ACME6_PROJECT;
    public Settings _settings;

    public DuplicateProjectNamesHealthCheck(IACME6_PROJECT PROJECT, IOptions<Settings> settings)
    {
        _ACME6_PROJECT = PROJECT;
        _settings = settings.Value;
    }

    public async Task<HealthCheckResult> CheckHealthAsync(HealthCheckContext context, CancellationToken cancellationToken = default)
    {
        List<string> duplicateProjectNames = await _ACME6_PROJECT.GetDuplicateProjectNamesAsync();

        if (duplicateProjectNames.Count > 0)
        {
            if (_settings.AutoFixDuplicateProjectNames)
            {                    
                await _ACME6_PROJECT.FixDuplicateProjectNamesAsync();                    
            }
            else
            {
                string errorMessage = "Duplicate Project Names found: ";
                foreach (string projectName in duplicateProjectNames)
                {
                    errorMessage += projectName + ") |";
                }
                errorMessage += "To autofix this set AutoFixDuplicateProjectNames to true in settings.";
                return new HealthCheckResult(status: context.Registration.FailureStatus, errorMessage);
            }
        }
        return HealthCheckResult.Healthy("OK");
    }
}

}

---

and the annotated line (see above) 236 complete error message as requested:

System.InvalidOperationException: A second operation was started on this context before a previous operation completed. This is usually caused by different threads concurrently using the same instance of DbContext. For more information on how to avoid threading issues with DbContext, see https://go.microsoft.com/fwlink/?linkid=2097913.
   at Microsoft.EntityFrameworkCore.Internal.ConcurrencyDetector.EnterCriticalSection()
   at Microsoft.EntityFrameworkCore.Query.Internal.SingleQueryingEnumerable`1.AsyncEnumerator.MoveNextAsync()
   at Microsoft.EntityFrameworkCore.Query.ShapedQueryCompilingExpressionVisitor.SingleAsync[TSource](IAsyncEnumerable`1 asyncEnumerable, CancellationToken cancellationToken)
   at Microsoft.EntityFrameworkCore.Query.ShapedQueryCompilingExpressionVisitor.SingleAsync[TSource](IAsyncEnumerable`1 asyncEnumerable, CancellationToken cancellationToken)
   at ACME.Org.Project.Services.ACME6.ACME6_PROJECT.GetDuplicateProjectNamesAsync() in E:\ACME\repo\acme-7-api\ACME.Org.Project\Services\ACME6\ACME6_PROJECT\ACME6_PROJECT.cs:line 236
   at ACME.Org.Project.Healthchecks.DuplicateProjectNamesHealthCheck.CheckHealthAsync(HealthCheckContext context, CancellationToken cancellationToken) in E:\ACME\repo\acme-7-api\ACME.Org.Project\Healthchecks\DuplicateProjectNamesHealthCheck.cs:line 23
   at Microsoft.Extensions.Diagnostics.HealthChecks.DefaultHealthCheckService.RunCheckAsync(IServiceScope scope, HealthCheckRegistration registration, CancellationToken cancellationToken)
Microsoft.Extensions.Diagnostics.HealthChecks.DefaultHealthCheckService: Error: Health check "Duplicate Project Names" threw an unhandled exception after 562.888ms

Answer 1

The reason for this is that AddCheck will create and use a single instance of the healthcheck type

return builder.Add(new HealthCheckRegistration(name,
     s => ActivatorUtilities.GetServiceOrCreateInstance<T>(s), 
     failureStatus, tags, timeout));

ActivatorUtilities.GetServiceOrCreateInstance<T>(s) will try to retrieve an instance of the type from DI or creates a new one. In either case, this means that DuplicateTaskNamesHealthCheck behaves as a singleton and any DbContext instance that gets injected into it will remain active until shutdown.

To solve this, the health check's constructor should accept either IServiceProvider or IDbContextFactory and create context instances whenever needed.

For example:

public class ExampleHealthCheck : IHealthCheck
{
   private readonly IDbContextFactory<ApplicationDbContext> _contextFactory;

   public ExampleHealthCheck(IDbContextFactory<ApplicationDbContext> contextFactory)
   {
       _contextFactory = contextFactory;
   }

   public Task<HealthCheckResult> CheckHealthAsync(HealthCheckContext context,
        CancellationToken cancellationToken = default(CancellationToken))
   {
       using (var context = _contextFactory.CreateDbContext())
       {
           // ...
       }
   }
}

IServiceProvider should be used to use a scoped or transient service like IACME6_PROJECT. IServiceProvider can be used to either retrieve transient services directly or create a scope to work with scoped services :

public class DuplicateProjectNamesHealthCheck : IHealthCheck
{
    public IServiceProvider _serviceProvider;
    public Settings _settings;

    public DuplicateProjectNamesHealthCheck(IServiceProvider serviceProvider, IOptions<Settings> settings)
    {
        _serviceProvider=serviceProvider;
        _settings = settings.Value;
    }

    public Task<HealthCheckResult> CheckHealthAsync(HealthCheckContext context,
        CancellationToken cancellationToken = default(CancellationToken))
    {
        using (var scope = _serviceProvider.CreateScope())
        {
            var project = scope.ServiceProvider.GetRequiredService< IACME6_PROJECT>();

            var duplicateProjectNames = await _ACME6_PROJECT.GetDuplicateProjectNamesAsync();
            ...
        }
    }