1

For quite awhile now I've read/believed that meta-programming unit tests is an anti-pattern. After all, tests are supposed to be DAMP not DRY right? So converting:

it('successfully does x', function() {
    assert(foo.x());
});
it('successfully does y' // ...
it('successfully does z',// ...

to:

var types = ['x', 'y', 'z'];
for (var i = 0; i < types.length; i++) {
    var type = types[i];
    it('successfully does ' + type, function() {
        assert(foo[type]());
    });
}

should be a bad idea ... right?

The thing is, I recently had a junior programmer write some unit tests in this meta-programming way, and I can't explain why those tests are problematic.

Normally I'd explain that it's hard to tell what exactly is failing in meta-programmed tests ... but in this programmer's tests its perfectly obvious because they modify the test label every time (ie. it says "successfully did x", so you know exactly what broke).

I'd also explain that it makes the test code itself messy ... but it's hard to argue that a simple foo[type] statement is more messy than repeating the entire contents of the test multiple times.

So, my question is two-part:

  1. Is meta-programming tests always a bad practice?
  2. If not, what criteria determine whether meta-programming a test is ok? Is it just "it's fine as long as the test code and output is clear", or are the other criteria to consider?

P.S. Please understand that I'm asking for an answer based on established unit testing best practices (which I believe is a legitimate SO question), NOT for subjective opinions (which would not be legitimate).

machineghost
  • 33,529
  • 30
  • 159
  • 234
  • 1
    Usually, there is no simple function that generates all the test cases. The test cases are things you make up and write out by hand. Usually, you don't really want repetitive test cases. But in cases where you decide that you do, and there is a simple function to generate the test cases, I don't see why using a for loop to make the tests would be a bad practice. If it's just, your programmer want to make an array and iterate over the array of test cases, instead of invoking the test function manually each time... I guess I would say he's making the code more complex for no gain. YMMV – Chris Beck Sep 14 '15 at 19:26
  • If the answer to "am I missing anything about standard test best practices that would make meta-tests bad?", is just "no" ... well then that sounds like an answer to me :) If you care to rephrase your comment as an answer I'll happily accept it. – machineghost Sep 15 '15 at 17:20

1 Answers1

1

Sometimes your code will have universal rules, based on what it inherits from etc. If you can cover code, even as new is added, then that is a test that requires less maintenance and is therefore - in my book - a better test.

Think about it. Instead of writing 5000 tests that ensure the same rule is followed on 100+ implementations, then you could simply have a single test that verifies it and it would even catch new things that were written. It is a means to programmatically ensure that existing as well as new code sticks to rules defined.

Meta-tests are a great way of testing in my book and it is the definition of "working smarter, not harder". They tend to be much harder to explain, so emphasis should be put on explaining what the test is doing as a lot of it is reflection. I see developers put everything together into a single method, without comments. I would instead see it a better way to use several methods, with correct naming, to help explain the narrative; of course with comments where you deem it relevant.

user2564050
  • 11
  • 1