Is JUnit the right tool to write performance tests?

Question

In the last week I've created two classes which my team expressed some concerns about in regards to their performance. To evaluate my code I wrote some simple JUnit tests which exercised these classes by building rich sets of test data, then feeding that data through the relevant methods for thousands of iterations. I recorded the runtime of each iteration then logged out the high, low, and average times by using loops and System.nanoTime(). Finally I had JUnit assert that the high and average times were within acceptable limits. This testing approach gave my team confidence in this code.

Is JUnit the right tool to test performance in this way? Are there better tools to test performance at the unit (method and class) level?

Answer 1

There may be better approaches, but there are also some frameworks that help implement benchmarking with JUnit. Some useful practices are warmup runs, statistical evaluations. Take a look at JUnitBenchmarks and JUnitPerf

EDIT looks like JUnitBenchmarks has been deprecated since the question has been stated. The project's maintainers recommend moving on to JMH. Thank you, Barry NL for the heads-up.

Answer 2

No. JUnit is designed for unit testing. There are lot of things that need to be taken care while writing performance tests in Java. Use Google Caliper which is specifically designed for writing micro-benchmark tests.

Look at How do I write a correct micro-benchmark in Java?

Answer 3

Your tests should be better defined as benchmark tests. Yes, JUnit can be used this way although it is not the best choice. But you can for example define max. evaluation time for test, so if algorithm is changed that caused performance degradation the test fails. Use @Test(timeout=12345) for configuration.

If you need real performance test think about JMeter.

Answer 4

JUnit is more designed for unit testing, maybe TestNG will be a better choice in this case, in particular, its @Dataprovider feature:

//This method will provide data to any test method that declares that its Data Provider
//is named "test1"
@DataProvider(name = "test1")
public Object[][] createData1() {
 return new Object[][] {
   { "Cedric", new Integer(36) },
   { "Anne", new Integer(37)},
 };
}
 
//This test method declares that its data should be supplied by the Data Provider
//named "test1"
@Test(dataProvider = "test1")
public void verifyData1(String n1, Integer n2) {
 System.out.println(n1 + " " + n2);
}

Here is the specific doc.