What form of testing should I perform?

Question

I want to write an algorithm (a bunch of machine learning algorithms) in C/C++ or maybe in Java, possibly in Python. The language doesn't really matter to me - I'm familiar with all of the above.

What matters to me is the testing. I want to train my models using training data. So I have the test input and I know what the output should be and I compare it to the model's output. What kind of test is this? Is it a unit test? How do I approach the problem? I can see that I can write some code to check what I need checking but I want to separate testing from main code. Testing is a well developed field and I've seen this done before but I don't know the name and type of this particular kind of testing so that I can read up on it and not create a mess. I'd be grateful if you could let me know what this testing method is called.

score 0 · Answer 1 · edited May 23 '17 at 12:12

0

This sounds a lot like Test-Driven Development (TDD), where you create unit-tests ahead of the production code. There are many detailed answers around this site on both topics. I've linked to a couple of pertinent questions to get you started.

edited May 23 '17 at 12:12

Community

1
1

answered Apr 23 '12 at 23:51

johnsyweb

136,902
23
188
247

score 0 · Answer 2 · answered Apr 23 '12 at 23:54

If your inputs/outputs are at the external interfaces of your full program, that's black box system testing. If you are going inside your program to zoom in on a particular function, e.g., a search function, providing inputs directly into the function and observing the behavior, that's unit testing. This could be done at function level and/or module level.

score 0 · Answer 3 · answered Apr 24 '12 at 00:00

If you're writing a machine learning project, the testing and training process isn't really Test-Driven Development. Have you ever heard of co-evolution? You have a set puzzles for your learning system that are, themselves, evolving. Their fitness is determined by how much they confound your cases.

For example, I want to evolve a sorting network. My learning system is the programs that produce networks. My co-evolution system generates inputs that are difficult to sort. The sorting networks are rewarded for producing correct sorts and the co-evolutionary systems are rewarded for how many failures they trigger in the sorting networks.

I've done this with genetic programming projects and it worked quite well.

score 0 · Answer 4 · answered Apr 24 '12 at 01:01

Probably back testing, which means you have some historical inputs and run your algorithm over them to evaluate the performance of your algorithm. The term you used yourself - training data - is more general and you could search for that to find some useful links.

Jupaol · Accepted Answer · 2012-04-24T01:38:28.543

Your best bet is watch the psychology of testing videos from the tetsing God http://misko.hevery.com/

Link of Misko videos:

http://misko.hevery.com/presentations/

And read this Google testing guide http://misko.hevery.com/code-reviewers-guide/

Edited:

Anyone can write tests, they are really simple and there is no magic to write a test, you can simply do something like:

var sut = new MyObject();
var res = sut.IsValid();
if(res != true)
{
  throw new ApplicationException("message");
}

That is the theory of course these days we have tools to simplify the tests and we can write something like this:

new MyObject().IsValid().Should().BeTrue();

But what you should do is focus on writing testable code, that's the magic key

Just follow the psychology of testing videos from Misko to get you started

score 0 · Answer 6 · answered Aug 14 '13 at 11:42

Its Unit testing. the controllers are tested and the code is checked in and out without really messing up your development code. This process is also called a Test Driven Development(TDD) where your every development cycle is tested before going into the next software iteration or phase.

score 0 · Answer 7 · answered Dec 13 '18 at 16:39

Although this is a very old post, my 2 cents :)

Once you've decided which algorithmic method to use (your "evaluation protocol", so to say) and tested your algorithm on unitary edge cases, you might be interested in ways to run your algorithm on several datasets and assert that the results are above a certain threshold (individually, or on average, etc.)

This tutorial explains how to do it within the pytest framework, that is the most popular testing framework within python. It is based on an example (comparing polynomial fitting algorithms on several datasets).

(I'm the author, feel free to provide feedback on the github page!)

What form of testing should I perform?

7 Answers7