How do I write a nose2 plugin that separates different types of tests?

Question

I'm writing a plugin that will separate treat my unit tests, functional tests & integration tests differently.
My tests folder will have the following structure exactly:

/tests
-- /unit
-- /functional
-- /integration

Each unit test will reside in the unit directory and each functional test will reside in the functional directory and so on.

I am familiar with the Layers plugin but I'd rather have my tests follow a convention.
Which hook exactly should I use to inject the appropriate Layer before tests are run? Should it be the loadTestsFromModule hook? Can you show me an example?

I'd also like to separate the summary report for each type of test.
Which hook should I use?

Answer 1

I got this working with nose2 by using the nose2 attrib plugin for discovery and some code copied from the nose1 attrib plugin which allowed me to decorate my tests.

Using the nose2 attrib plugin

You will see the nose2 attrib plugin allows for custom attributes to be defined on test functions and classes.

For this to work, you have to specify the attributes of the tests after defining the test function.

class MyTestCase(unittest.TestCase):
    def test_function(self):
        self.assertEqual(1+1, 2)
    test_function.custom_attr1 = True
    test_function.custom_attr2 = ['foo', 'bar']

Then you can run a set of filtered tests by specifying -A or --attribute as a nose2 command-line argument to list the attribute(s) you to match against your test suite. You can even use the expression command-line argument of -E or --eval-attribute which allows more complex Python expressions for matching test attributes.

e.g. nose2 -v -A custom_attr1
will run all tests which have a custom_attr1 specified with a truthy value.

Using decorators to specify test attributes

This wasn't quite good enough for me though because I didn't like the idea of defining these attributes on tests after their definition. I wanted to use a decorator instead but nose2 didn't have a built-in decorator for doing this.

I went to the nose1 source code for its attrib plugin and copied the source for the attr function.

def attr(*args, **kwargs):
    """Decorator that adds attributes to classes or functions
    for use with the Attribute (-a) plugin.
    """
    def wrap_ob(ob):
        for name in args:
            setattr(ob, name, True)
        for name, value in kwargs.iteritems():
            setattr(ob, name, value)
        return ob
    return wrap_ob

I put this into a test/attrib_util.py file. Now I can specify attributes using the decorator instead. My original test class code from above can be converted to the (IMO) simpler:

from test.attrib_util import attr

class MyTestCase(unittest.TestCase):
    @attr('custom_attr1', custom_attr2=['foo', 'bar'])
    def test_function(self):
        self.assertEqual(1+1, 2)

You will notice that the attributes can be specified as either args or kwargs; all args will get a default value of True.

You can also even use this attr decorator on a test class or base class and the attributes will be applied to all test functions defined within. This allows for very easy separation of unit and functional tests.

from test.attrib_util import attr

@attr('functional')
class FunctionalTestCase(unittest.TestCase):
    pass

class MyFunctionalCase(FunctionalTestCase):
    def test_function(self):
        print 'this will be considered a "functional" test function'

Answer 2

You don't need to write a plug-in, the built-in attr module is designed for this purpose. It does not depend on your file hierarchy, however. Instead, you mark individual tests as unit, functional, or integration. This would look like:

from nose.plugins import attrib

@attrib.attr("functional")
class FunctionalTestCase(unittest.TestCase):
    pass

To run only the functional tests, you would then do:

nosetests -a functional

If I were creating this test layout, I would probably have 3 unittest.TestCase subclasses, already marked with "unit", "functional", and "integration". New tests could easily inherit the proper test type.

Answer 3

If you already have the tests sorted into directories (as you mentioned), you could write a plugin that uses the wantDirectory method.

import os.path
from nose.plugins import Plugin

class TestCategory(Plugin):
    """
    Run tests in a defined category (unittest, functional, integration.  Always
    runs uncategorized tests.
    """
    def wantDirectory(self, dirname):
         dirname = os.path.basename(dirname)
         if (dirname in ('unit', 'functional', 'integration') and 
             dirname != self.category):
             return False
         return None

You will want to write options() and configure() methods for this plug-in to deal with enabling and disabling it and gleaning the user's choice of category. When running nosetests you would choose from the three categories:

nosetests --category functional

Since only one test category is run at a time, you would get a separate report for each test category. You could always, of course, run all tests by not enabling this plugin.

(adding as a different answer because it is a completely different approach).