Timed nose tests not failing properly

Question

I have the following test that does not fail when running an especially long fib assert.

Tests that don't fail properly

#!/usr/env/bin python2.7

import unittest
from fib import fib
from nose.tools import timed


def test_gen(expected, actual):
    @timed(.001)
    def test_method(self):
        return self.assertEqual(expected, actual)
    return test_method

if __name__ == '__main__':
    all_cases = {
        'user': ((fib(40), 102334155), (fib(2), 1), (fib(5), 5)),
    }
    fails = {}
    for username, cases in all_cases.items():
        class FibTests(unittest.TestCase):
            pass

        for index, case in enumerate(cases):
            test_name = 'test_{0}_{1}'.format(username, index)
            test = test_gen(case[1], case[0])
            setattr(FibTests, test_name, test)

        suite = unittest.TestLoader().loadTestsFromTestCase(FibTests)
        result = unittest.TextTestRunner(verbosity=2).run(suite)
        fails[username] = len(result.failures)
    print fails

(Slow) Fib.py Implementation

def fib(x):
    if x == 0:
        return 0
    elif x == 1:
        return 1

    return fib(x - 2) + fib(x - 1)

Tests that fail properly

import unittest
from fib import fib
from nose.tools import timed


def test_gen(expected, actual):
    @timed(.001)
    def test_method(self):
        time.sleep(.2)
        return self.assertEqual(expected, actual)
    return test_method

Reading the code, it looks like you're only testing timing on the `assertEqual`, not on the actual calculation of the number. Is this what you want? You might need to put the call to `fib` in your `test_method` somehow. — John Lyon, Jan 17 '13 at 23:41

Silas Ray · Accepted Answer · 2013-01-18T01:54:27.557

You are timing the wrong thing, and never actually calling your test method. You are also going to an awful lot of effort to dynamically create and add your cases to your class that does nothing but act as a container for tests when nose supports generator test cases, which would be much easier to read and follow than what you have here. Also, is this a test file or a piece of product code? If it's a test file, then having all of that code in if __name__ == '__main__' is kind of odd, and if it is a product code file, then having the test_gen function and the unittest and nose import statements in the uncoditionally run part doesn't make much sense. I'd recommend doing it the following way, and not trying to make the test script self-runnable; just launch it with nose.

from fib import fib
from nose.tools import timed

fib = timed(.001)(fib)

def execute(username, fib_arg, expected_output):
    result = fib(fib_arg)
    assert result == expected_output, ('%s fib(%d) got %d, expected %d'
                                       % (username, fib_arg, result, expected_output))

def test_fib():
    for name, datasets in (('user', ((40, 102334155), (2, 1), (5, 5))),):
        for arg, expected in datasets:
            yield execute, name, arg, expected

Yup it was pretty complicated, got that code from: http://stackoverflow.com/questions/14372287/python-unittests-continue-after-failure/14374179#14374179 but like your solution much better. — prafulfillment, Jan 18 '13 at 01:51
There was an error in the assertion failure string construction, but the way `timed` is used is valid. Decorators are just syntactical sugar shorthand for what I did here; any decorator can be called as a function like I did here. They will expect and return a callable. Rewrapping your test function in yet another function just so you can decorate the new one just creates inefficient duplicate function wrapping. — Silas Ray, Jan 18 '13 at 01:56
Totally understand what you're saying but if you look at the timed decorator, it doesn't return the result from the function. It's fine, I patched the method and submitted a pull request on Github: https://github.com/nose-devs/nose/pull/602 — prafulfillment, Jan 18 '13 at 02:43

Timed nose tests not failing properly

Tests that don't fail properly

(Slow) Fib.py Implementation

Tests that fail properly

1 Answers1