A Nose plugin to specify the order of unit test execution

Question

I have a desire to use Nose for an over the wire integration test suite. However, the order of execution of some of these tests is important.

That said, I thought I would toss together a quick plugin to decorate a test with the order I want it executed: https://gist.github.com/Redsz/5736166

def Foo(unittest.TestCase):

    @step(number=1)
    def test_foo(self):
        pass

    @step(number=2)
    def test_boo(self):
        pass

From reviewing the built in plugins I had thought, I could simply override loadTestsFromTestCase and order the tests by the decorated 'step number'.:

def loadTestsFromTestCase(self, cls):
    """
    Return tests in this test case class. Ordered by the step definitions.
    """
    l = loader.TestLoader()
    tmp = l.loadTestsFromTestCase(cls)

    test_order = []
    for test in tmp._tests:
        order = test.test._testMethodName
        func = getattr(cls, test.test._testMethodName)
        if hasattr(func, 'number'):
            order = getattr(func, 'number')
        test_order.append((test, order))
    test_order.sort(key=lambda tup: tup[1])
    tmp._tests = (t[0] for t in test_order)
    return tmp

This method is returning the tests in the order I desire, however when the tests are being executed by nose they are not being executed in this order?

Perhaps I need to move this concept of ordering to a different location?

UPDATE: As per the comment I made, the plugin is actually working as expected. I was mistaken to trust the pycharm test reporter. The tests are running as expected. Rather than removing the question I figured I would leave it up.

Well, I guess this question can be disregarded, the plugin is working as expected...I had a brain fart while running it from pychamr where the test reporter was being ordered differently. However the tests are actually being ordered as expected during execution. — Jesse, Jun 08 '13 at 20:12
I don't want to presume too much, since I don't know your code base, but based on my experience with automated testing, it's very likely that the best long term solution is re-architecting your tests in a way such that they no longer have dependencies on each other. You might consider repeating necessary preconditions in setup functions, or for sequences of events that really are dependent, merging them into a single test. Also keep in mind that, as a unit test, you should generally be testing individual "units" of functionality, and avoiding external IO wherever possible. — GrandOpener, Aug 03 '13 at 16:19

score 20 · Answer 1 · answered Sep 05 '13 at 02:57

From the documentation:

[...] nose runs functional tests in the order in which they appear in the module file. TestCase-derived tests and other test classes are run in alphabetical order.

So a simple solution might be to rename the tests in your test case:

class Foo(unittest.TestCase):

    def test_01_foo(self):
        pass

    def test_02_boo(self):
        pass

score 0 · Answer 2 · answered May 07 '15 at 17:52

I found a solution for it using PyTest ordering plugin provided here.

Try py.test YourModuleName.py -vv in CLI and the test will run in the order they have appeared in your module (first test_foo and then test_bar)

I did the same thing and works fine for me.

Note: You need to install PyTest package and import it.

A Nose plugin to specify the order of unit test execution

2 Answers2

Linked