1

When I run my xUnit unit tests I sometimes get an error message like "Transaction (Process ID 58) was deadlocked on lock resources with another process and has been chosen as the deadlock victim" on one or more of the tests, seemingly randomly. If I re-run any failing test on its own it passes.

What should I do to prevent this? Is there an option to run the tests one-after-another instead of all at once?

(N.B. I'm running the tests over the API methods in my ASP.Net 5 MVC controllers under Visual Studio 2015)

Here's an example of one of my occasionally failing tests:

[Fact]
private void TestREAD()
{
    Linq2SQLTestHelpers.SQLCommands.AddCollections(TestCollections.Select(collection => Convert.Collection2DB(collection)).ToList(), TestSettings.LocalConnectionString);
    foreach (var testCollection in TestCollections)
    {
        var testCollectionFromDB = CollectionsController.Get(testCollection.Id);
        Assert.Equal(testCollection.Description, testCollectionFromDB.Description);
        Assert.Equal(testCollection.Id, testCollectionFromDB.Id);
        Assert.Equal(testCollection.IsPublic, testCollectionFromDB.IsPublic);
        Assert.Equal(testCollection.LayoutSettings, testCollectionFromDB.LayoutSettings);
        Assert.Equal(testCollection.Name, testCollectionFromDB.Name);
        Assert.Equal(testCollection.UserId, testCollectionFromDB.UserId);
    }
}

There are two methods the test calls, here's the controller method:

[HttpGet("{id}")]
public Collection Get(Guid id)
{
    var sql = @"SELECT * FROM Collections WHERE id = @id";
    using (var connection = new SqlConnection(ConnectionString))
    {
        var collection = connection.Query<Collection>(sql, new { id = id }).First();
        return collection;
    }
}

and here's the helper method:

public static void AddCollections(List<Collection> collections, string connectionString)
{
    using (var db = new DataClassesDataContext(connectionString))
    {
        db.Collections.InsertAllOnSubmit(collections);
        db.SubmitChanges();
    }
}

(Note that I'm using Dapper as the micro-ORM in the controller method and so, to avoid potentially duplicating errors in the test, I'm using LINQ to SQL instead in the test to set-up and clean-up test data.)

There are also database calls in the unit test's class's constructor and Dispose method. I can add them to the post if needed.

dumbledad
  • 16,305
  • 23
  • 120
  • 273
  • 1
    You might want to share the kind of test you're writing - some insight into your `async` / `Task` management etc. Right now you could be doing anything and obviously there are some people who are having success, we just need to figure out what's leaking/locking in your context... – Ruben Bartelink Feb 05 '16 at 10:21
  • Done, is that enough? – dumbledad Feb 05 '16 at 10:28
  • 1
    Hmmm - doesn't look like much that can be blamed on xUnit there. Are you leraking connections [and their associated locks] ? – Ruben Bartelink Feb 05 '16 at 10:33
  • I'll add the controller method, I don't think so. – dumbledad Feb 05 '16 at 10:34

1 Answers1

1

OK, so looks like a plain vanilla case of deadlocks in your app and the need to handle that - what is your plan on the app side?

The tests and their data rigging can potentially fall prey to the same thing. xUnit doesnt have anything to address this and I'd strongly argue it shouldnt.

So in both the test and the app, you need failure/retry management.

For a web app, you have a fire them a picture of a whale and let them try again pattern but ultimately you want a real solution.

For a test, you don't want whales and definitely want to handle it, i.e. not be brittle.

I'd be using Poly to wrap retry decoration around anything in either the app or the tests that's prone to significant failures -- your exercise is to figure out what are the significant failures in your context.


Under normal circumstances a database with a single reader/writer operating synchronously shouldn't deadlock. Analysing why it happens is a matter of doing the analysis on the DB side. The tools that side would also likely quickly reveal to you if e.g. you have some aspect of your overall System Under Test which is resulting in competing work.

(Obviously your snippets are incomplete as there is a disconnect between CollectionsController.Get(testCollection.Id) and the fact that the controller method is not static - the point of this discussion should not be down at that level IMO though)

Ruben Bartelink
  • 59,778
  • 26
  • 187
  • 249
  • There's very little initialization in the controller's constructor, I just pass in a connection string wrapped as an IOptions for dependency injection. The tests seem to fail randomly, so they are all "prone to significant failures". I'm not sure what tools are at my disposal to analyze the deadlocks on the DB side but I'll investigate. Thanks. – dumbledad Feb 05 '16 at 13:08
  • @dumbledad SQL Profiler can show them. Re "The tests seem to fail randomly," one thing to figure out is whether its the test fixture setup or the stuff in the controller that's causing the issue - i.e. if its the actual controller, then you could look on it as a bug in the controller which needs to be fixed by having some constrained retrying in there – Ruben Bartelink Feb 05 '16 at 13:56
  • 1
    Profiler showed up a deadlock that seemed to be coming from my INSTEAD OF DELETE trigger that cascaded the deletes. So I think you were correct, that was a deadlock that may have bitten me in live code, not just a test. I ended up following the advice [here](http://stackoverflow.com/a/35361855/575530) and removing the trigger and rewriting it as a stored procedure and now my tests all run without error. – dumbledad Feb 15 '16 at 17:35
  • Interesting. I'm a bit meh on replacing the trigger with SPs - whatever way you've structured and sequenced the dependencies that made the system prone to deadlock in the first instance are still at large... – Ruben Bartelink Feb 15 '16 at 21:26
  • Maybe sequencing is key. In my DELETE_USER trigger I deleted the necessary Items and Collections and relied on the DELETE_ITEM and DELETE_COLLECTION triggers to mop up. In my Delete_User stored procedure I go through all the tables, leaves then branches, and do the deletes. I do not call the Delete_Item nor the Delete_Collection stored procedures. – dumbledad Feb 16 '16 at 07:29
  • @dumbledad Yes, that's the sort of thing it's likely to be. Have a search for lock ordering deadlock and you'll find a good explanation – Ruben Bartelink Feb 16 '16 at 09:05