Deadlocks causing 'Server failed to resume the transaction' with NHibernate and distributed transactions

Question

We are having an issue when using NHibernate with distributed transactions.

Consider the following snippet:

//
// There is already an ambient distributed transaction
//
using(var scope = new TransactionScope()) {
    using(var session = _sessionFactory.OpenSession())
    using(session.BeginTransaction()) {
        using(var cmd = new SqlCommand(_simpleUpdateQuery, (SqlConnection)session.Connection)) {
            cmd.ExecuteNonQuery();
        }

        session.Save(new SomeEntity());
        session.Transaction.Commit();
    }
    scope.Complete();
}

Sometimes, when the server is under extreme load, we'll see the following:

1. The query executed with cmd.ExecuteNonQuery is chosen as a deadlock victim (we can see it in SQL Profiler), but no exception is raised.
2. session.Save fails with the error message, "The operation is not valid for the state of the transaction."
3. Every time this code is executed after that, session.BeginTransaction fails. The first few times, the inner exception varies (sometimes it is the deadlock exception that should have been raised in step 1). Eventually it stabilizes to "The server failed to resume the transaction. Desc:3800000177." or "New request is not allowed to start because it should come with valid transaction descriptor."

If left alone, the application will eventually (after seconds or minutes) recover from this condition.

Why is the deadlock exception not being reported in step 1? And if we can't resolve that, then how can we prevent our application from temporarily becoming unusable?

The issue has been reproduced in the following environments

• Windows 7 x64 and Windows Server 2003 x86
• SQL Server 2005 and 2008
• .NET 4.0 and 3.5
• NHibernate 3.2, 3.1 and 2.1.2

I've created a test fixture which will sometimes reproduce the issue for us. It is available here: http://wikiupload.com/EWJIGAECG9SQDMZ

Answer 1

We've finally narrowed this down to a cause.

When opening a session, if there is an ambient distributed transaction, NHibernate attaches an event handler to the Transaction.TransactionCompleted, which closes the session when the distributed transaction is completed. This appears to be subject to a race condition wherein the connection may be closed and returned to the pool before the deadlock error propagates across, leaving the connection in an unusable state.

The following code will reproduce the error for us occasionally, even without any load on the server. If there is extreme load on the server, it becomes more consistent.

using(var scope = new TransactionScope()) {
    //
    // Force promotion to distributed transaction
    //
    TransactionInterop.GetTransmitterPropagationToken(Transaction.Current);

    var connection = new SqlConnection(_connectionString);
    connection.Open();

    //
    // Close the connection once the distributed transaction is
    // completed.
    //
    Transaction.Current.TransactionCompleted += 
        (sender, e) => connection.Close();

    using(connection.BeginTransaction())
        //
        // Deadlocks but sometimes does not raise exception
        //
        ForceDeadlockOnConnection(connection);

    scope.Complete();
}

//
// Subsequent attempts to open a connection with the same
// connection string will fail
//

We have not settled on a solution, but the following things will eliminate the problem (while possibly having other consequences):

• Turning off connection pooling
• Using NHibernate's AdoNetTransactionFactory instead of AdoNetWithDistributedTransactionFactory
• Adding error handling that calls SqlConnection.ClearPool() when the "server failed to resume the transaction" error occurs

According to Microsoft (https://connect.microsoft.com/VisualStudio/feedback/details/722659/), the SqlConnection class is not thread-safe, and that includes closing the connection on a separate thread. Based on this response we have filed a bug report for NHibernate (http://nhibernate.jira.com/browse/NH-3023).

Answer 2

It is an NHibernate issue. NHibernate is not opening and closing the connection on the same thread, which is not supported by ADO.NET. You can work around it by opening and closing the connection yourself. NHibernate will not close the connection unless it has also opened it.

Workaround

var connection = ((SessionFactoryImpl)_sessionFactory).ConnectionProvider.GetConnection();
using(var session = _sessionFactory.OpenSession(connection))
{
   //do database stuff
}
connection.Close();

Answer 3

not a definitive answer, but i suspect you have some problems with session management and that you are using the same session across multiple calls to handlers. i don't think it's actually the connection that is in a bad state, but rather the nhibernate session. this doesn't seem to jive with you not seeing the problem with connection pooling turned off, so i may be off base, but i still suspect it has to do with reusing sessions.

the first thing i would suggest is that you try to confirm this by logging the hashcode of the session and the hashcode of session.GetSessionImplementation() (my understanding of using the castle nhibernate facility is that you will see the same instance of session, even though it is actually a different session and the session implementation will actually show a difference). see if you are seeing the same hashcodes being used in handling different messages.

if it is a question of session management, try using a nservicebus module to manage your sessions for your handlers. here is a post from andreas about doing that. i don't think his edit about having a way to do this built in on the trunk was in the 2.5 release, so you probably want to go ahead with this. (i could be wrong about that.)

http://andreasohlund.net/2010/02/03/nhibernate-session-management-in-nservicebus/

Answer 4

This doesn't exactly solve your problem, but you could make your IPreInsertEventListener just send a NSB message, and then have the receiver of the message invoke the stored procedure. I've done that with problematic pre-and post event listeners while using NHibernate and NSB in the past.

Another thought is have your pre-event listener create its own connection object wrapped in a nice using statement, then it won't touch NHibernate's connection. If it deadlocks, then just do a throw an make sure you've disposed of any object's in scope.