3

Original

About 45 days ago I updated a project I have that was on .NET 5 to .NET 6. It's using ASP.NET Core 6 and Entity Framework Core 6 across eight websites. All eight websites use a shared configuration process and get their configurations from AWS Parameter Store.

Since the update, there are random exceptions happening with the following message:

The specified transaction is not associated with the current connection. Only transactions associated with the current connection may be used.

Retrying the request right after usually resolves the issue, but not always. I don't remember having any such issues under .NET 5, ASP.NET Core 5 and Entity Framework Core 5. I'm not sure how to resolve this. Here is how I'm setting up and configuring EF, does anyone notice anything that could be the cause? The code didn't really change between 5 and 6...:

public static IServiceCollection AddApplicationDbContextPool(
    this IServiceCollection services,
    IConfig config,
    int poolSize = 1024) {
    if (config is null) {
        throw new ArgumentNullException(nameof(config));
    }

    services.AddDbContextPool<ApplicationContext>(
        _ => {
            _.UseSqlServer(config.DefaultConnection,
                           __ => {
                               __.UseNetTopologySuite();

                               if (!config.IsDevelopment) {
                                   return;
                               }

                               __.CommandTimeout(int.MaxValue);
                           });

            if (!config.IsDevelopment) {
                return;
            }

            _.EnableDetailedErrors()
             .EnableSensitiveDataLogging();
        }, poolSize);

    services.AddDbContextPool<SerilogContext>(
        _ => {
            _.UseSqlServer(config.SerilogConnection,
                           __ => {
                               __.UseNetTopologySuite();// may be unnecessary, but keeping until I verify.

                               if (!config.IsDevelopment) {
                                   return;
                               }

                               __.CommandTimeout(int.MaxValue);
                           });

            if (!config.IsDevelopment) {
                return;
            }

            _.EnableDetailedErrors()
             .EnableSensitiveDataLogging();
        });

    NtsGeometryServices.Instance = new NtsGeometryServices(
        NetTopologySuite.Geometries.Implementation.CoordinateArraySequenceFactory.Instance,
        new PrecisionModel(1000d),
        4326,
        GeometryOverlay.NG,
        new CoordinateEqualityComparer()
    );

    return services;
}

The backing database is SQL Server 2019. Not sure if it matters, but with the update the database server was swapped with a new instance that was more tightly tuned and configured than the previous one and is also on faster hardware than the previous one. I doubt it's the database that's the issue, something about the applications interacting with it is.

There is one app that's actually using Dapper to interact with the database since it's the hottest app in the group. Here is the Dapper configuration, and the specific app is using a Singleton scope (I need to remove the transient, nothing is using it anymore):

public static IServiceCollection AddApplicationDapper(
    this IServiceCollection services,
    IConfig config,
    ServiceLifetime serviceLifetime = ServiceLifetime.Transient) {
    if (config is null) {
        throw new ArgumentNullException(nameof(config));
    }

    SqlMapper.AddTypeHandler(new GeometryHandler<Point>(true));
    SqlMapper.AddTypeHandler(new GeometryHandler<Polygon>(true));

    return serviceLifetime == ServiceLifetime.Singleton
        ? services.AddSingleton(
            _ => new SqlConnection(config.DefaultConnection))
        : services.AddTransient(
            _ => new SqlConnection(config.DefaultConnection));
}

Could it be the use of async/await throughout the apps? I do use .ConfigureAwait(false) pretty much everywhere, could that thread transfer be the cause? This also didn't change much between 5 and 6.

I thought it was the EF Core Triggered package and have since removed it, but the exceptions continue to be thrown. Help will be highly appreciated!

Update #1

@Dai

To clarify the Dapper usage with a singleton SqlConnection instance, it is only used in one of the apps, which is technically an ASP.NET Core app for common management with the other ASP.NET Core apps, but it doesn't listen to web traffic at all. Instead it is an IoT listener and handler where it registers a couple of IHostedServices classes as singletons. The IoT services listen for reports from specific devices then pass it up to a handling service. Only the handling service is given a SqlConnection instance, so in this specific case it is a 1-to-1 use, so a singleton SqlConnection works fine. This is how this project has been since the days of .NET Core 2.0.

It's not this project that's getting the exceptions, it is the other ones using EF Core to interact with the database that are. I doubt a single instance of SqlConnection used in a single handler service for the lifetime of the IoT app is the cause.

@Gert Arnold

I'm 99.9991% sure that I am awaiting everywhere. At the very least ReSharper will yell at me if I don't.

Update #2

System.InvalidOperationException: The specified transaction is not associated with the current connection. Only transactions associated with the current connection may be used.
    at Microsoft.EntityFrameworkCore.Storage.RelationalTransaction..ctor(IRelationalConnection connection, DbTransaction transaction, Guid transactionId, IDiagnosticsLogger`1 logger, Boolean transactionOwned, ISqlGenerationHelper sqlGenerationHelper)
    at Microsoft.EntityFrameworkCore.SqlServer.Storage.Internal.SqlServerTransactionFactory.Create(IRelationalConnection connection, DbTransaction transaction, Guid transactionId, IDiagnosticsLogger`1 logger, Boolean transactionOwned)
    at Microsoft.EntityFrameworkCore.Storage.RelationalConnection.BeginTransactionAsync(IsolationLevel isolationLevel, CancellationToken cancellationToken)
    at Microsoft.EntityFrameworkCore.Storage.RelationalConnection.BeginTransactionAsync(CancellationToken cancellationToken)
    at Microsoft.EntityFrameworkCore.Update.Internal.BatchExecutor.ExecuteAsync(IEnumerable`1 commandBatches, IRelationalConnection connection, CancellationToken cancellationToken)
    at Microsoft.EntityFrameworkCore.Update.Internal.BatchExecutor.ExecuteAsync(IEnumerable`1 commandBatches, IRelationalConnection connection, CancellationToken cancellationToken)
    at Microsoft.EntityFrameworkCore.Update.Internal.BatchExecutor.ExecuteAsync(IEnumerable`1 commandBatches, IRelationalConnection connection, CancellationToken cancellationToken)
    at Microsoft.EntityFrameworkCore.ChangeTracking.Internal.StateManager.SaveChangesAsync(IList`1 entriesToSave, CancellationToken cancellationToken)
    at Microsoft.EntityFrameworkCore.ChangeTracking.Internal.StateManager.SaveChangesAsync(StateManager stateManager, Boolean acceptAllChangesOnSuccess, CancellationToken cancellationToken)
    at Microsoft.EntityFrameworkCore.SqlServer.Storage.Internal.SqlServerExecutionStrategy.ExecuteAsync[TState,TResult](TState state, Func`4 operation, Func`4 verifySucceeded, CancellationToken cancellationToken)
    at Microsoft.EntityFrameworkCore.DbContext.SaveChangesAsync(Boolean acceptAllChangesOnSuccess, CancellationToken cancellationToken)
    at Microsoft.EntityFrameworkCore.DbContext.SaveChangesAsync(Boolean acceptAllChangesOnSuccess, CancellationToken cancellationToken)
    at REDACTED.App.Devices.Locate.CommandHandler.Handle(Command command, CancellationToken cancellationToken) in E:\Software Development\REDACTED\REDACTED.App\Features\Devices\Locate.cs:line 69
Gup3rSuR4c
  • 9,145
  • 10
  • 68
  • 126
  • What's with the crazy indenting? – Dai May 08 '23 at 17:22
  • 1
    _"Here is the Dapper configuration, and the specific app is using a Singleton scope...."_ **DO NOT DO THAT**: `SqlConnection` objects are not intended to be shared nor long-life'd: in a DI system `SqlConnection` objects should not be registered as a service _at all_ (instead use a factory service) - or at the very least should be registered as transient or scoped (and the scope's lifetime shouldn't be more than a few minutes - and prohibit its use as a ctor-injected transient dependency in a long-life'd or singleton type). – Dai May 08 '23 at 17:24
  • See here: https://stackoverflow.com/questions/42937942/dapper-with-net-core-injected-sqlconnection-lifetime-scope – Dai May 08 '23 at 17:24
  • I've updated my question to clarify my use of the singleton `SqlConnection` with Dapper. – Gup3rSuR4c May 08 '23 at 18:11
  • 1
    Do you know if any of your project's code is using `TransactionScope` at all? (Or worse: the horrible, evil "_ambient_ transactions" feature drempt up by clearly the most sadistic person ever employed at Microsoft... excepting the designer of the Windows 11 Start Menu. – Dai May 08 '23 at 18:12
  • sorry, ot - but I had to vote for this "the most sadistic person ever .. excepting the designer of the Windows 11 Start Menu" – Christoph Lütjen May 08 '23 at 18:18
  • No, I'm not explicitly using `TransactionScope` anywhere, nor do I use `DbContext.Database.BeginTransaction()`. Whatever EF Core does internally for transactions, that's it, so ambient transactions from EFC? That's why I was wondering about the `ConfigureAwait(false)` use and how it can jump around on threads. – Gup3rSuR4c May 08 '23 at 18:19
  • 1
    @Gup3rSuR4c Can you show us any of the exceptions' stack-traces at all? Are you able to get a full dump of the process when that exception is thrown/caught too? That way you can examine async-task states even when the exception's call-stack is almost empty due to resuming in a thread-pool thread. – Dai May 08 '23 at 19:10
  • @Dai Sorry for the delay, I had to leave for work. I've posted what I have on the exception. Can't really do a dump because its random and transient. Its only happened to me once in development (and reloading the page worked as expected), but seems to happen several times a day in production. – Gup3rSuR4c May 09 '23 at 04:39
  • @Gup3rSuR4c You can configure .NET in production systems to make a dump when some CLR event happens, such as a particular exception being thrown: https://learn.microsoft.com/en-us/dotnet/core/diagnostics/dumps – Dai May 09 '23 at 04:59
  • @Gup3rSuR4c Also this: https://stackoverflow.com/questions/1134048/generating-net-crash-dumps-automatically – Dai May 09 '23 at 05:00
  • @Gup3rSuR4c Can you explain what `App.Devices.Locate.CommandHandler.Handle` does and how it works? When I see names like "Locate" and "CommandHandler" I instantly think of early (2005-ish) `static`-based IoC libs, frameworks (e.g. `ServiceLocator`) - which is a plausible explanation for why unrelated code suddenly finds itself in enlisted in an ambient transaction scope it didn't ask for. – Dai May 09 '23 at 09:44
  • It's just a MediatR handler. On an unrelated note, I had to update some records last night, and ran into the exception after the 3rd record. I then decided to swap out of the `AddDbContextPool()` to just `AddDbContext()` and the remaining 22 updates worked as expected. Haven't seen transaction exceptions in the logs since that update either. Will keep an eye on it today. Could the DbContext pool not have been cleaning up and resetting the DbContext instances correctly before reuse? – Gup3rSuR4c May 09 '23 at 13:53

0 Answers0