How to execute code only on test failures with python unittest2?

Question

I have some class-based unit tests running in python's unittest2 framework. We're using Selenium WebDriver, which has a convenient save_screenshot() method. I'd like to grab a screenshot in tearDown() for every test failure, to reduce the time spent debugging why a test failed.

However, I can't find any way to run code on test failures only. tearDown() is called regardless of whether the test succeeds, and I don't want to clutter our filesystem with hundreds of browser screenshots for tests that succeeded.

How would you approach this?

score 8 · Accepted Answer · answered Sep 06 '12 at 00:25

8

Found a solution - I can override failureException:

@property
def failureException(self):
    class MyFailureException(AssertionError):
        def __init__(self_, *args, **kwargs):
            self.b.save_screenshot('%s.png' % self.id())
            return super(MyFailureException, self_).__init__(*args, **kwargs)
    MyFailureException.__name__ = AssertionError.__name__
    return MyFailureException

This seems incredibly hacky but it seems to work so far.

answered Sep 06 '12 at 00:25

craigds

2,072
14
25

Wow, what a hack. An exception class, wrapped in a function, wrapped in a property, all to get the `TestCase` instance into the exception's `__init__()` method. Ugly, but at least it works! – kindall Sep 06 '12 at 15:39

score 3 · Answer 2 · edited May 23 '17 at 12:18

Here is similar approach to @craigds answer, but with directory support and better compatibility with Python 3:

@property
def failureException(self):
    class MyFailureException(AssertionError):
        def __init__(self_, *args, **kwargs):
            screenshot_dir = 'reports/screenshots'
            if not os.path.exists(screenshot_dir):
                os.makedirs(screenshot_dir)
            self.driver.save_screenshot('{0}/{1}.png'.format(screenshot_dir, self.id()))
            return super(MyFailureException, self_).__init__(*args, **kwargs)
    MyFailureException.__name__ = AssertionError.__name__
    return MyFailureException

This was actually found in this blog.

I've extended it further more with argparse:

parser.add_argument("-r", "--reports-dir", action="store",   dest="dir",      help="Directory to save screenshots.", default="reports")

so the dir can be specified dynamically either by system variable or passed argument:

screenshot_dir = os.environ.get('REPORTS_DIR', self.args.dir) + '/screenshots'

This is especially useful, if you've additional wrapper to run all your scripts, like a base class.

score 2 · Answer 3 · answered Sep 05 '12 at 22:17

2

Override fail() to generate the screenshot and then call TestCase.fail(self)?

answered Sep 05 '12 at 22:17

kindall

178,883
35
278
309

1

That only helps if the testcase fails because someone called `self.fail()`. Other failures, such as `self.assertTrue()`, ... etc bypass this. – craigds Sep 05 '12 at 23:59
Really? I can't believe they didn't have those call `fail()`. Of course, you could also override all those other methods (sigh). – kindall Sep 06 '12 at 00:14
Some of them do, but others don't. Seems a little crazy ;) – craigds Sep 06 '12 at 00:26

score 2 · Answer 4 · answered May 31 '13 at 15:23

sys.exc_info() should give you exit information on whether a test failed or not. So something like this:

def tearDown(self):
    if sys.exc_info()[0]:
        path = os.path.join(os.path.dirname(os.path.abspath(__file__)), '../failures', self.driver.browser)
        if not os.path.exists(path):
            try:
                os.makedirs(path)
            except Exception:
                # Since this might not be thread safe
                pass
        filename = '%s.%s.png' % (self.__class__.__name__, self._testMethodName)
        file_path = os.path.join(path, filename)
        self.driver.get_screenshot_as_file(file_path)

score 1 · Answer 5 · edited May 23 '17 at 11:47

Use a decorator around each test.

The safest way to remember to decorate new tests, or to avoid going back and decorating a bunch of existing tests, is to use a metaclass to wrap all of the test functions. The How to wrap every method of a class? answer provides the basics of what you need.

You probably should filter the functions that are wrapped down to just the tests, e.g.:

class ScreenshotMetaClass(type):
    """Wraps all tests with screenshot_on_error"""
    def __new__(meta, classname, bases, classDict):
        newClassDict = {}
        for attributeName, attribute in classDict.items():
            if type(attribute) == FunctionType and 'test' in attributeName.lower():
                # replace the function with a wrapped version
                attribute = screenshot_on_error(attribute)
            newClassDict[attributeName] = attribute
        return type.__new__(meta, classname, bases, newClassDict)

How to execute code only on test failures with python unittest2?

5 Answers5

Linked

Related