How do I test for speed with PyTest / tox?

Question

For testing machine learning algorithms / repositories, I see three things that matter:

1. Does it crash
2. Does it have a minimum test accuracy
3. Is it fast enough

While (1) and maybe (2) is standard unit testing, I'm not too sure how to deal with (3). Can I test this with pytest / tox?

I found pytest-benchmark, but how would I do this for example for lidtk?

In pseudo-code, I want to do the following:

def classifier_predict(input_features):
    # do something smart, but maybe too time-consuming
    return result

def input_generator():
    # Generate something random which classifier_predict
    # can work on - don't measure this time!
    return input_features

class Agents(unittest.TestCase):
    def test_classifier_predict():
        self.assertMaxTime(classifier_predict,
                           input_generator,
                           max_time_in_ms=100)

Answer 1

Handcraft-solution

Here is the pseudo-code of a rather hand-crafted solution:

def classifier_predict(input_features):
    # do something smart, but maybe too time-consuming
    return result

def input_generator():
    # Generate something random which classifier_predict
    # can work on - don't measure this time!
    return input_features

class Agents(unittest.TestCase):
    def test_classifier_predict():
        nb_tests = 1000
        total_time = 0.0
        for _ in range(nb_tests):
            input_ = input_generator()
            t0 = time.time()
            classifier_predict(input_)
            t1 = time.time()
            total_time += t1 - t0
        self.assertLessEqual(total_time / nb_tests, 100)

Drawbacks

• No nice graphs (like pytest-benchmark seems to generate)
• In general, hard limits might be difficult due to different hardware and also due to different external work load