0

How can a method be tested with random input in a systematic way? What I mean by this is, if the input data for the units being tested changes each run, how can you effectively find out which values caused the failure and retest those same values after the code has been change? For example the function int[] foo(int[] data) and it can be checked if the return value is correct or not, after writing test cases for "hard coded inputs" (e.g. {1,2,3}) how can random input be tested? If it's random each time, any errors wont be reproducible. I could print the random input to the screen, but that would become messy for each function. Any suggestions?

Also, is it still unit testing if the whole program is being tested? For example calling the constructor and all methods of class in one @Test? The class only has one public method to begin with so this can't really be avoided, but I'm wondering if JUnit is the best tool for it.

Celeritas
  • 14,489
  • 36
  • 113
  • 194
  • when the input is random, how do you know what the expected result is? – Timothy Truckle Oct 19 '16 at 06:41
  • @TimothyTruckle in some situations it can be checked, for example it's easy to check if the value returned is sorted. – Celeritas Oct 19 '16 at 06:50
  • It’s a debate where I work, we certainly don’t agree. In some cases randomized tests are the best you can do, and among some people they are in high esteem. For testing a median algorithm I wrote a test that shuffled the input, but since I knew what the elements were, I could also tell the correct median independently of their order. Obviously this test did not stand alone, but it may in theory some day catch a yet undisvored error. – Ole V.V. Oct 19 '16 at 07:08
  • For reproduceability, print you random seed to the screen. – Ole V.V. Oct 19 '16 at 07:11
  • When we UnitTest whe want to fix certain behavior in a way that when a test fails it is clear to understan why. with Random Input you cannot (easily) reproduce the error condition. – Timothy Truckle Oct 19 '16 at 07:55
  • Read about (automated) "regression testing". And "random testing". And "unit testing" vs "integration testing" & "black-box testing" vs "white-box testing". And just "software testing" & "program testing". Also *information hiding* & *design by contract*. – philipxy Jul 20 '17 at 21:58

2 Answers2

1

How can a method be tested with random input in a systematic way?

In the general case and in the most strict sense, you cannot. By "strict sense" I mean: validating the correctness of the output no matter which input is presented.

Assuming it would be possible, 'strictness' imply that your test case can compute the result of the function on each and every (random) input. Which means you will need to write a piece of code that replicates the method to be tested - theoretically possible, but leading to a paradoxical situation:

  • assume you find multiple bugs in the method under test. What is the cheapest way to correct it? Of course, substituting the method code with the testing code

  • and now the tester (the author of the currently implemented method) needs to... what?... write another "incarnation" of the function in order to test her own implementation?

However, "fuzzing" is still a valid method: except that it is never to be taken in the strict sense; the tests expects the results to expose certain traits/invariants/etc, something that can be defined/checked not matter what the input is. For example, "the method never throws", or "the returned array has the same length as the input array (or double the size, or all elements are odd, or whatever" or "the result is alway a proper HTML page which passes the W3C markup validator".

tested with random input in a systematic way?

You almost have an oxymoron here, mate, like "honest politician" or "aeroplane-safe 2016-made Galaxy 7 Note". If testing is "systematic" it means "there is a (rule) system that governs the way the tests are conducted" - almost quite opposite to "random input".

The trick to reconcile the two: you still have a (rule-based) system to categorize your input (e.g. equivalence partitioning ), except that instead of taking a certain (constant) representative for your categories, you pick the representatives at random. I'm going to repeat this: inside each categories, you pick your representative at random; (as opposed to "pick a random input and see which category it belongs to").
How's this useful? Not much added value, because according to the system/equivalence, picking a representative is as good as picking any other.

Sorry my QA mate, you don't get off-the-hook in regards with your responsibility as a tester to design and plan for the tests (no matter if you use random techniques in generating your input).

If it's random each time, any errors wont be reproducible. I could print the random input to the screen, but that would become messy for each function. Any suggestions?

This is a frivolous reason to avoid random input if random input is deemed necessary: just use some tools to visually organize the testing logs if the simple flowing text format is so hard to read.

E.g. output your test log formatted as JSON with a certain structure and use/write a visualisation tool to represent/explore/fold/unfold it in such a way that human exploration is not a pain in the nether back part of the body.
If your job is to automate the testing, it is supposed you are able to code, isn't it?

Also, is it still unit testing if the whole program is being tested?

"Whole program is being tested" exactly how? What is the specific goal of this "whole system testing"?
There is a distinction between "unit testing" (even a more-than-comprehensive 10000% coverage unit testing) and "functional testing" or "integration testing" or "performance testing" or "usability testing" or "localization testing" (this including the gra'ma' of the UI/error messages) - all the latter belonging to "whole program testing" without being unit testing.

Hint: in defining the type of testing, the specific goal one has in mind when designing/executing test takes precedence over the means used in testing; I've seen testing performed manually using GUI test harnesses, in which the testers where manually writing values to unit test the SDK underneath.

On the other side, there are categories of non-unit testing which may make use of unit testing techniques (e.g. a Web or REST Service - wrap and present it as a proxy-function API and then you can write your tests using JUnit/TestBG or whatever unit test framework you fancy. And yet, you are doing functional or integration testing).

Adrian Colomitchi
  • 3,974
  • 1
  • 14
  • 23
0

Property-based testing could be a solution for you. Basically, the idea is to have a framework generating all sorts of input data (random, edge cases) which is then fed into your implementation.

It's true that with random data you may end up with test cases behaving differently with every run. But at least the test framework usually would show you which input is used when tests are failing and you can have a further look into the reasons for failing. It's not a guarantee that your method will work in 100% of the cases, but at least you get some coverage and it's still better than nothing.

Typically, such a framework also allows you to restrict the generated data to a set which makes sense for your implementation. Or you can implement your own generator providing data for your tests.

For Java there is e.g. JUnit-Quickcheck which integrates with JUnit: http://pholser.github.io/junit-quickcheck/site/0.6.1/

A lot has already been written about the differences between unit tests/integration tests etc. Maybe have a look here: What is the difference between integration and unit tests?

Community
  • 1
  • 1
Josef
  • 321
  • 1
  • 6