EntityFrameworkCore - Diagnosing "The connection was not closed. The connection's current state is open."

Question

We seem to have done something to introduce some kind of threading issue into our app. It's a dotnet core 1.1.2 application, using EntityFrameworkCore. Intermittently, but reproducably, we will get one of these errors:

System.InvalidOperationException: 'The connection was not closed. The connection's current state is open.' exception when running an EF query.

or

System.InvalidCastException: 'Unable to cast object of type 'System.Data.ProviderBase.DbConnectionClosedConnecting' to type 'System.Data.SqlClient.SqlInternalConnectionTds'.'

or

System.ObjectDisposedException: 'Cannot access a disposed object. A common cause of this error is disposing a context that was resolved from dependency injection and then later trying to use the same context instance elsewhere in your application. This may occur if you are calling Dispose() on the context, or wrapping the context in a using statement. If you are using dependency injection, you should let the dependency injection container take care of disposing context instances.'

This can happen at almost any point in the app, but this method seems to be the most frequent culprit (this does get called somewhat frequently):

public class WidgetConfigurationRepository : IWidgetConfigurationRepository
{

    public WidgetConfigurationRepository(LocalContext dataContext)
    {
        Context = dataContext ?? throw new ArgumentNullException(nameof(dataContext));
    }

    private LocalContext _context { get; private set; }

    public async Task<WidgetConfiguration> LoadConfigurationAsync(Guid widgetId)
    {
        return await _context.WidgetConfigurations
            .Include(x => x.Options)
            .FirstOrDefaultAsync(x => x.WidgetId.Equals(widgetId));
    }
...
}

the repository is created via the Core DI container, and is injected at runtime:

services.AddScoped<IWidgetConfigurationRepository, WidgetConfigurationRepository>();

The Repository and the LocalContext are both registered as Scoped, and there are no Singleton accessors of the LocalContext.

Walking through the stack trace (and the parallel stacks) shows that every async method is awaited, and I have gone through replacing the interfaces in the application with the implementations, to hopefully try to find any async without await.

The LocalContext itself is created via a LocalDbContextFactory that is also registered as Scoped. It reads data from a central data context, checks some data and then instantiates a new instance of the LocalContext. This is only happening once per request, as expected.

Ultimately, I'm looking for some help working out what could be causing this, our app is now fairly large and I'm not sure I can provide enough code snippets to help.

At the moment, I think my best option is to write a Roslyn analyzer to go through all the methods that return Task or Task<T> and check that something happens with the return object, but I'm wondering if there is something easier that I may have missed.

There is this related question: EF. The connection was not closed. The connection's current state is connecting with the

1. Re-check that IUserService is registered with "scope" lifetime, and all it dependencies (userManager, dbContext) too
2. Do not use IServiceProvider you obtained during app startup for scope-bases services resolution - it is NOT related to current request scope and return instances from "some other universe". Use HttpContext.RequestServices for service resolution.
3. Check that your are "awaiting" all async methods. If you start second request while still executing first one - you may possibly "catch" dbContext during "connecting" stage.
4. Your JwtMessageHandler instance is one/single per app. So don't use it's property for storing _userService (remove private IUserService _userService). Instead, use local variable inside OnMessageReceived (var _userService = ...).

As far as I can tell, we are not being caught by any of these, almost all of our services are Scoped or Transient.

We use ServiceProvider.GetService<>() during the app startup, to resolve the database to run Migrations on, but not beyond that.

I think it's possible that there is an async without an await, but I don't seem to be able to find it, and we aren't getting compiler warnings for it.