NUnit Order Attribute does not work with Parallizable

Question

How can I take advantage of Parallelizable tests while using the Order Attribute?

Nunit v3.9.0

I realize the amount of other posts related to this topic... but no ordering seems to work when ParallelScope.Self is used.

The reason I need to order my test execution. I am running Selenium tests on a remote EC2 and the ChromeDriver has a bug where it hangs. [Bug] [Bug] [Bug] [Bug]

I need to run some initial tests that can fail silently to ensure the ChromeDriver is running correctly... THEN I need the remainder of the tests to run in parallel.self to take advantage of the speed.

My "initiation tests" are very very basic. I run three of these in their own [TestFixture]

[TestFixture]
[Order(1)]
public class Initiate
{
    [TestCase(TestName = "001")]
    public void Initiate_ChromeDriver_2()
    {
        try
        {
            IWebDriver driver = new ChromeDriver();
            driver.Navigate().GoToUrl(<url>);
            driver.WaitForPageLoad();
            driver.Quit();
            driver.Dispose();
        }
        catch (Exception ex)
        {
            Assert.Inconclusive(ex.Message);
        }
        Assert.Pass();
    }
}

*driver.WaitForPageLoad() is an extension.

Output:

But... the fixtures run in a completely random order under the Parallelizable assembly, which ignores the order attribute.

I've read so many posts. But I cannot seem to get my tests to order correctly.
I've tried:

[Test, Order(1)] as per the documentation
[Catagory("yada")] ref
[TestCase(TestName = "001")]
[TestFixture][Order(1)] added in v3.8
Alphabetizing test names and fixture names

Thanks in advance!

EDIT: ~~This is a known issue:~~ #2521
Any workarounds?

EDIT 2: Not an issue. Read Charlie's comments below.

Note that issue 2521 is unconfirmed, waiting for a repro case from the submitter. Hence, nobody knows if it's really a problem and nobody is working on it. :-( — Charlie, Mar 02 '18 at 23:22

score 3 · Accepted Answer · answered Mar 02 '18 at 23:42

The OrderAttribute controls the order in which tests are started WRT other tests at the same level. In the case of your example, the fixture will start to run before any other fixtures in the same namespace.

The attribute on the fixture has no impact on how test cases within the fixture are run, of course.

The Order feature gives complete ordering of tests only if there is no parallelism involved. With a single thread of execution, the test that starts first has to finish before the next one can start. But with multiple threads of execution, that is no longer true.

Lets say you have three tests at the same level with order values of 1, 2 and 3. If there are three or more threads of execution, all three may start: 1, 2, 3! They run in parallel.

IOW, the simple ordering provided by NUnit is not a dependency. That is, there is no guarantee that the test with Order 1 will complete before the next test runs. If you want that, it's a requested feature, but not yet implemented.

OTOH, it doesn't look like you really want test ordering. What you are trying to do looks to me like a job for a one-time setup method rather than a test. In fact, the dependencies among setups, tests and teardowns is currently the only dependency that exists in NUnit, so you should try to formulate your dependency requirements along the lines of the setup model.

In your case, I believe you want a SetUpFixture or maybe several. The fixture can be placed outside all namespaces to give you something that runs before anything else in the assembly or in a particular namespace if the tests that require it are all in one namespace. It might look something like this:

[SetUpFixture]
public class Initiate
{
  [OneTimeSetUp]
  public void Initiate_ChromeDriver_2()
  {
        IWebDriver driver = new ChromeDriver();
        driver.Navigate().GoToUrl(<url>);
        driver.WaitForPageLoad();
        driver.Quit();
        driver.Dispose();
  }
}

I left in all your code, although I'm not sure I understand why you want to close and dispose the driver. Don't you need it for your tests? The disposal could be moved to a OneTimeTearDown method.

As always, thank you so much for answering my NUnit questions so quickly. That SetUpFixture will work great! — J-Roel, Mar 06 '18 at 18:07
After further investigation I found a DNS server error that was causing my problems. There are a few threads[1](https://github.com/SeleniumHQ/docker-selenium/issues/222) , and of course I can't find the one I was thinking of. But, regarding this issue as it relates to selenium: ChromeDriver calls the googleapis end point. Not sure why it needs to do this. You can shut it off by returning a falsey IP in your hosts file. Or you can simply refresh your ChromeDriver (driver.Navigate().Refresh() in c#). — J-Roel, Mar 06 '18 at 18:15
I setup a sort of brute force page wait extension. It will check the page loading, if it hangs for more than 10 seconds. Then it refreshes the page and checks it again. It repeats this up to four times. I then call that from the [SetUpFixture] as instructed by Charlie. Works great. — J-Roel, Mar 06 '18 at 18:18

NUnit Order Attribute does not work with Parallizable

1 Answers1