Running test script scenarios in python

Question

I'm developing a script to test another program. Basically I replicated a class with all the functions that the program I am testing have. To make things simple, assume I have only 2 functions: set(value) and add(value). set(value) sets an accumulator value and add(value) adds value to it; and I'm testing a program that should do the same thing.

I also have a check() function that verifies that the accumulator value in which set(value) and add(value) operates have the same value as the value retrieved by the program I'm testing; so, after EACH operation, I want to run this check() function and make sure they match.

To try to make this clear, here's a (non-tested) example:

def set(self, value):
    self.acc = value
    sendCommandToMyProgram('set', value)

def add(self, value):
    self.acc += value
    sendCommandToMyProgram('add', value)

def check(self):
    return self.acc == sendCommandToMyProgram('check', value)

and I want to do "test scenarios" like:

set(1)
check()
add(2)
check()
add(-2)
check()
add(3)

and many others

What would be the best way to make this "test scenario" script? It would be nice if it could work with functions that takes different parameters (in this example, both set() and add() received just one parameter - value -, but this isn't always the case).

thanks

Answer 1

Assuming that you're not wanting to check the value all the time during normal operation, the thing that pops to mind first would be the built-in unit test module.

If you are needing to do more complex things that you cannot cleanly isolate the test as shown below, check out a some of my issues I had to work around. It's worth noting that I ended up rolling my own test runner that behaves very similar to unit-test, but with greater control. I'll be pushing it up to py-pi in the next month or so.

Working Unit Test Examples

import unittest

class RealAccumlator(object):
    def __init__(self):
        self.acc = 0

    def add(self, val):
        self.acc += val

    def set(self, val):
        self.acc = val

    def get(self):
        return self.acc


# Using a global variable as I dont know how "sendCommandToMyProgram"
# is used
real_acc = RealAccumlator()


# Test Script

class MyAccumlator(object):
    def __init__(self):
        self.acc = 0

    def add(self, val):
        self.acc += val
        real_acc.add(val)

    def set(self, val):
        self.acc = val
        real_acc.set(val)

    def check(self):
        return self.acc == real_acc.get()

class MockedAccumlatorTests(unittest.TestCase):

    def setUp(self):
        self.acc = MyAccumlator()

    def test_SetFunction(self):
        test_values = range(-10,10)

        # Test the set commands
        for val in test_values:
            self.acc.set(val)
            self.assertTrue(self.acc.check())

    def test_AddFunction(self):
        test_values = range(-10,10)

        # Set the acc to a known value and to a quick test
        self.acc.set(0) 
        self.assertTrue(self.acc.check())

        # Test the add commands
        for val in test_values:
            self.acc.add(val)
            self.assertTrue(self.acc.check())


class RealAccumlatorTests(unittest.TestCase):

    def test_SetFunction(self):
        test_values = range(-10,10)

        # Test the set commands
        for val in test_values:
            real_acc.set(val)
            self.assertEqual(val, real_acc.get())

    def test_AddFunction(self):
        test_values = range(-10,10)

        # Set the acc to a known value and to a quick test
        real_acc.set(0) 
        self.assertEqual(0, real_acc.get())

        # Test the add commands
        expected_value = 0
        for val in test_values:
            expected_value += val
            real_acc.add(val)
            self.assertEqual(expected_value, real_acc.get())

if __name__ == '__main__':
    unittest.main()

Update based on accepted answer

If this is only going to be a test script and your mock accumulator isn't used out side this tests consider the following mod. I still believe writing the unittests will serve you better in the long run

class MyAccumlator(object):
    def __init__(self):
        self.acc = 0

    def add(self, val):
        self.acc += val
        real_acc.add(val)
        assert(self.check())
        return self.check()

    def set(self, val):
        self.acc = val
        real_acc.set(val)
        assert(self.check())
        return self.check()

    def check(self):
        return self.acc == real_acc.get()

This would allow you to iterate over whatever lists you want without having to think about calling the check function. There are two methods provided that you could check.

1. Leaving the assert call would throw an exception if they ever didn't match (only suggesting under the assumption this will live as a test script).

2. Removing the assert and checking the return status from your calling script (rather than an explicit call to check() might clean things up as well.

Answer 2

I needed to run the "check()" after each operation.

I ended up creating "test scenarios" like:

test_scenarios =
    [
        [
            (obj.set, {1}),
            (obj.add, {5}),
            (obj.add, {-2})
        ],
        [
            (obj.set, {-2}),
            (obj.add, {3}),
            (obj.add, {-5})
            (obj.add, {1})
        ]
    ]

then I iterate over the test scenarios basically calling test[0](**test[1]) and then check()

maybe not the best solution, but the best I could think of

Answer 3

Set the value. Create a list with the values to add. Then, loop through that list, running check and add with each loop. i.e.: numlist = [1, 2, -2, 3]; for num in numlist: obj.check(); obj.add(num)