NUnit Test Run Order

Question

By default nunit tests run alphabetically. Does anyone know of any way to set the execution order? Does an attribute exist for this?

Answer 1

I just want to point out that while most of the responders assumed these were unit tests, the question did not specify that they were.

nUnit is a great tool that can be used for a variety of testing situations. I can see appropriate reasons for wanting to control test order.

In those situations I have had to resort to incorporating a run order into the test name. It would be great to be able to specify run order using an attribute.

Answer 2

NUnit 3.2.0 added an OrderAttribute, see:

https://github.com/nunit/docs/wiki/Order-Attribute

Example:

public class MyFixture
{
    [Test, Order(1)]
    public void TestA() { ... }


    [Test, Order(2)]
    public void TestB() { ... }

    [Test]
    public void TestC() { ... }
}

Answer 3

Your unit tests should each be able to run independently and stand alone. If they satisfy this criterion then the order does not matter.

There are occasions however where you will want to run certain tests first. A typical example is in a Continuous Integration situation where some tests are longer running than others. We use the category attribute so that we can run the tests which use mocking ahead of the tests which use the database.

i.e. put this at the start of your quick tests

[Category("QuickTests")]

Where you have tests which are dependant on certain environmental conditions, consider the TestFixtureSetUp and TestFixtureTearDown attributes, which allow you to mark methods to be executed before and after your tests.

Answer 4

Wanting the tests to run in a specific order does not mean that the tests are dependent on each other - I'm working on a TDD project at the moment, and being a good TDDer I've mocked/stubbed everything, but it would make it more readable if I could specify the order which the tests results are displayed - thematically instead of alphabetically. So far the only thing I can think of is to prepend a_ b_ c_ to classes to classes, namespaces and methods. (Not nice) I think a [TestOrderAttribute] attribute would be nice - not stricly followed by the framework, but a hint so we can achieve this

Answer 5

Regardless of whether or not Tests are order dependent... some of us just want to control everything, in an orderly fashion.

Unit tests are usually created in order of complexity. So, why shouldn't they also be run in order of complexity, or the order in which they were created?

Personally, I like to see the tests run in the order of which I created them. In TDD, each successive test is naturally going to be more complex, and take more time to run. I would rather see the simpler test fail first as it will be a better indicator as to the cause of the failure.

But, I can also see the benefit of running them in random order, especially if you want to test that your tests don't have any dependencies on other tests. How about adding an option to test runners to "Run Tests Randomly Until Stopped"?

Answer 6

I am testing with Selenium on a fairly complex web site and the whole suite of tests can run for more than a half hour, and I'm not near to covering the entire application yet. If I have to make sure that all previous forms are filled in correctly for each test, this adds a great deal of time, not just a small amount of time, to the overall test. If there's too much overhead to running the tests, people won't run them as often as they should.

So, I put them in order and depend on previous tests to have text boxes and such completed. I use Assert.Ignore() when the pre-conditions are not valid, but I need to have them running in order.

Answer 7

I really like the previous answer.

I changed it a little to be able to use an attribute to set the order range:

namespace SmiMobile.Web.Selenium.Tests
{
    using System;
    using System.Collections.Generic;
    using System.Linq;
    using System.Reflection;
    using NUnit.Framework;

    public class OrderedTestAttribute : Attribute
    {
        public int Order { get; set; }


        public OrderedTestAttribute(int order)
        {
            Order = order;
        }
    }

    public class TestStructure
    {
        public Action Test;
    }

    class Int
    {
        public int I;
    }

    [TestFixture]
    public class ControllingTestOrder
    {
        private static readonly Int MyInt = new Int();

        [TestFixtureSetUp]
        public void SetUp()
        {
            MyInt.I = 0;
        }

        [OrderedTest(0)]
        public void Test0()
        {
            Console.WriteLine("This is test zero");
            Assert.That(MyInt.I, Is.EqualTo(0));
        }

        [OrderedTest(2)]
        public void ATest0()
        {
            Console.WriteLine("This is test two");
            MyInt.I++; Assert.That(MyInt.I, Is.EqualTo(2));
        }


        [OrderedTest(1)]
        public void BTest0()
        {
            Console.WriteLine("This is test one");
            MyInt.I++; Assert.That(MyInt.I, Is.EqualTo(1));
        }

        [OrderedTest(3)]
        public void AAA()
        {
            Console.WriteLine("This is test three");
            MyInt.I++; Assert.That(MyInt.I, Is.EqualTo(3));
        }


        [TestCaseSource(sourceName: "TestSource")]
        public void MyTest(TestStructure test)
        {
            test.Test();
        }

        public IEnumerable<TestCaseData> TestSource
        {
            get
            {
                var assembly =Assembly.GetExecutingAssembly();
                Dictionary<int, List<MethodInfo>> methods = assembly
                    .GetTypes()
                    .SelectMany(x => x.GetMethods())
                    .Where(y => y.GetCustomAttributes().OfType<OrderedTestAttribute>().Any())
                    .GroupBy(z => z.GetCustomAttribute<OrderedTestAttribute>().Order)
                    .ToDictionary(gdc => gdc.Key, gdc => gdc.ToList());

                foreach (var order in methods.Keys.OrderBy(x => x))
                {
                    foreach (var methodInfo in methods[order])
                    {
                        MethodInfo info = methodInfo;
                        yield return new TestCaseData(
                            new TestStructure
                                {
                                    Test = () =>
                                        {
                                            object classInstance = Activator.CreateInstance(info.DeclaringType, null);
                                            info.Invoke(classInstance, null);
                                        }
                                }).SetName(methodInfo.Name);
                    }
                }

            }
        }
    }
}

Answer 8

I know this is a relatively old post, but here is another way to keep your test in order WITHOUT making the test names awkward. By using the TestCaseSource attribute and having the object you pass in have a delegate (Action), you can totally not only control the order but also name the test what it is.

This works because, according to the documentation, the items in the collection returned from the test source will always execute in the order they are listed.

Here is a demo from a presentation I'm giving tomorrow:

using System;
using System.Collections.Generic;
using System.Linq;
using System.Text;
using NUnit.Framework;

namespace NUnitTest
{
    public class TestStructure
    {
        public Action Test;
    }

    class Int
    {
        public int I;
    }

    [TestFixture]
    public class ControllingTestOrder
    {
        private static readonly Int MyInt= new Int();

        [TestFixtureSetUp]
        public void SetUp()
        {
            MyInt.I = 0;
        }

        [TestCaseSource(sourceName: "TestSource")]
        public void MyTest(TestStructure test)
        {
            test.Test();
        }

        public IEnumerable<TestCaseData> TestSource
        {
            get
            {
                yield return new TestCaseData(
                    new TestStructure
                    {
                        Test = () =>
                        {
                            Console.WriteLine("This is test one");
                            MyInt.I++; Assert.That(MyInt.I, Is.EqualTo(1));
                        }
                    }).SetName(@"Test One");
                yield return new TestCaseData(
                    new TestStructure
                    {
                        Test = () =>
                        {
                            Console.WriteLine("This is test two");
                            MyInt.I++; Assert.That(MyInt.I, Is.EqualTo(2));
                        }
                    }).SetName(@"Test Two");
                yield return new TestCaseData(
                    new TestStructure
                    {
                        Test = () =>
                        {
                            Console.WriteLine("This is test three");
                            MyInt.I++; Assert.That(MyInt.I, Is.EqualTo(3));
                        }
                    }).SetName(@"Test Three");
            }
        }
    }
}

Answer 9

I am working with Selenium WebDriver end-to-end UI test cases written in C#, which are run using NUnit framework. (Not unit cases as such)

These UI tests certainly depend on order of execution, as other test needs to add some data as a precondition. (It is not feasible to do the steps in every test)

Now, after adding the 10th test case, i see NUnit wants to run in this order: Test_1 Test_10 Test_2 Test_3 ..

So i guess i have to too alphabetisize the test case names for now, but it would be good to have this small feature of controlling execution order added to NUnit.

Answer 10

There are very good reasons to utilise a Test Ordering mechanism. Most of my own tests use good practice such as setup/teardown. Others require huge amounts of data setup, which can then be used to test a range of features. Up till now, I have used large tests to handle these (Selenium Webdriver) integration tests. However, I think the above suggested post on https://github.com/nunit/docs/wiki/Order-Attribute has a lot of merit. Here is an example of why ordering would be extremely valuable:

• Using Selenium Webdriver to run a test to download a report
• The state of the report (whether it's downloadable or not) is cached for 10 minutes
• That means, before every test I need to reset the report state and then wait up to 10 minutes before the state is confirmed to have changed, and then verify the report downloads correctly.
• The reports cannot be generated in a practical / timely fashion through mocking or any other mechanism within the test framework due to their complexity.

This 10 minute wait time slows down the test suite. When you multiply similar caching delays across a multitude of tests, it consumes a lot of time. Ordering tests could allow data setup to be done as a "Test" right at the beginning of the test suite, with tests relying on the cache to bust being executed toward the end of the test run.

Answer 11

This question is really old now, but for people who may reach this from searching, I took the excellent answers from user3275462 and PvtVandals / Rico and added them to a GitHub repository along with some of my own updates. I also created an associated blog post with some additional info you can look at for more info.

Hope this is helpful for you all. Also, I often like to use the Category attribute to differentiate my integration tests or other end-to-end tests from my actual unit tests. Others have pointed out that unit tests should not have an order dependency, but other test types often do, so this provides a nice way to only run the category of tests you want and also order those end-to-end tests.

Answer 12

If you are using [TestCase], the argument TestName provides a name for the test.

If not specified, a name is generated based on the method name and the arguments provided.

You can control the order of test execution as given below:

[Test]
[TestCase("value1", TestName = "ExpressionTest_1")]
[TestCase("value2", TestName = "ExpressionTest_2")]
[TestCase("value3", TestName = "ExpressionTest_3")]
public void ExpressionTest(string  v)
{
    //do your stuff
}

Here I used the method name "ExpressionTest" suffix with a number.

You can use any names ordered alphabetical see TestCase Attribute

Answer 13

Usually Unit Test should be independent, but if you must, then you can name your methods in alphabetical order ex:

[Test]
public void Add_Users(){}

[Test]
public void Add_UsersB(){}

[Test]
public void Process_Users(){}

or you can do..

        private void Add_Users(){}

        private void Add_UsersB(){}

        [Test]
        public void Process_Users()
        {
           Add_Users();
           Add_UsersB();
           // more code
        }

Answer 14

I'm suprised the NUnit community hasn't come up with anything, so I went to create something like this myself.

I'm currently developing an open-source library that allows you to order your tests with NUnit. You can order test fixtures and ordering "ordered test specifications" alike.

The library offers the following features:

• Build complex test ordering hierarchies
• Skip subsequent tests if an test in order fails
• Order your test methods by dependency instead of integer order
• Supports usage side-by-side with unordered tests. Unordered tests are executed first.

The library is actually inspired in how MSTest does test ordering with .orderedtest files. Please look at an example below.

[OrderedTestFixture]
public sealed class MyOrderedTestFixture : TestOrderingSpecification {
    protected override void DefineTestOrdering() {
        TestFixture<Fixture1>();

        OrderedTestSpecification<MyOtherOrderedTestFixture>();

        TestFixture<Fixture2>();
        TestFixture<Fixture3>();
    }

    protected override bool ContinueOnError => false; // Or true, if you want to continue even if a child test fails
}

Answer 15

You should not depend on the order in which the test framework picks tests for execution. Tests should be isolated and independent. In that they should not depend on some other test setting the stage for them or cleaning up after them. They should also produce the same result irrespective of the order of the execution of tests (for a given snapshot of the SUT)

I did a bit of googling. As usual, some people have resorted to sneaky tricks (instead of solving the underlying testability/design issue

• naming the tests in an alphabetically ordered manner such that tests appear in the order they 'need' to be executed. However NUnit may choose to change this behavior with a later release and then your tests would be hosed. Better check in the current NUnit binaries to Source Control.
• VS (IMHO encouraging the wrong behavior with their 'agile tools') has something called as "Ordered tests" in their MS testing framework. I didn't waste any time reading up but it seems to be targeted towards the same audience

See Also: characteristics of a good test

Answer 16

In case of using TestCaseSource the key is to override string ToString method, How that works:

Assume you have TestCase class

public class TestCase
{
    public string Name { get; set; }
    public int Input { get; set; }
    public int Expected { get; set; }
}

And a list of TestCases:

private static IEnumerable<TestCase> TestSource()
{
    return new List<TestCase>
    {
        new TestCase()
        {
           Name = "Test 1",
           Input = 2,
           Expected = 4
        },
        new TestCase()
        {
            Name = "Test 2",
            Input = 4,
            Expected = 16
        },
        new TestCase()
        {
            Name = "Test 3",
            Input = 10,
            Expected = 100
        }
    };
}

Now lets use it with a Test method and see what happen:

[TestCaseSource(nameof(TestSource))]
public void MethodXTest(TestCase testCase)
{
    var x = Power(testCase.Input);
    x.ShouldBe(testCase.Expected);
}

This will not test in order and the output will be like this:

So if we added override string ToString to our class like:

public class TestCase
{
    public string Name { get; set; }
    public int Input { get; set; }
    public int Expected { get; set; }

    public override string ToString()
    {
        return Name;
    }
}

The result will change and we get the order and name of test like:

Note:

1. This is just example to illustrate how to get name and order in test, the order is taken numerically/alphabetically so if you have more than ten test I suggest making Test 01, Test 02.... Test 10, Test 11 etc because if you make Test 1 and at some point Test 10 than the order will be Test 1, Test 10, Test 2 .. etc.
2. The input and Expected can be any type, string, object or custom class.
3. Beside order, the good thing here is that you see the test name which is more important.