4

I'm looking for a Maven plugin, which would measure runtime complexity of all unit tests in the module and fail the build if it's higher than the threshold. I need this to prevent accidental increases of complexity, which doesn't break the functionality but makes the code much slower. Does it exist?

yegor256
  • 102,010
  • 123
  • 446
  • 597
  • How do you define runtime complexity? And what code are you concerned about: your unit tests or production code. It is very common to have integration tests which fail if there is a performance regression. – tgdavies Dec 20 '21 at 05:31
  • @tgdavies It's up to the plugin how to define "runtime complexity," I'm ready to rely on their metric(s). Of course, it will test both test and production code together. How would you write such an integration test you mentioned? – yegor256 Dec 20 '21 at 05:34

2 Answers2

1

Complexity is about the change in performance measures as a particular parameter or parameters increase. The time to run a suite of unit tests doesn't have a meaningful complexity measure ... because:

  • there is (typically) no set of parameters to vary for any individual test, and
  • the overall performance of the test suite is an aggregate of the individual tests that typically do completely different things.

Methods have complexity. But you would need deep semantic knowledge of which methods matter, and how they are intended to be used, and what the parameters for the complexity measure should be in order to generate some meaningful "empirical" complexity approximations for them. I don't think there is much chance that anyone has tried to implement a Maven or IDE plugin to do do this kind of thing. Or that it would be reliable if they ever did.

My advice would be:

  • Work out the methods, and parameters that are critical to the performance of your application.

  • Write some performance tests to run the methods for a range of parameters.

  • Write some "harness" code to run the tests, capture the measures and compare them with the "predictions" of the complexity class you are aiming for ... for each measure.

  • Integrate these tests into your testing regime.

But beware of the traps of writing Java micro-benchmarks; see How do I write a correct micro-benchmark in Java?.

Actually ... my better advice would be to not attempt to measure complexity at all. Instead, just create some regular benchmarks for your application, and decide on some the performance thresholds to trigger your attention. After all, your real goal should be performance, not complexity.

Stephen C
  • 698,415
  • 94
  • 811
  • 1,216
  • I want to put this performance/complexity check into the build pipeline, preventing programmers from increasing the complexity. Just like we run all unit tests and style checkers. – yegor256 Dec 20 '21 at 09:01
  • You can put the benchmarks into the pipeline and have them fail the build if your performance thresholds are breached. But measuring >complexity< is not the way to go. – Stephen C Dec 20 '21 at 09:54
  • If I do what you suggest, my build will depend on the environment where it's being executed. On a faster computer it may not fail. On the other hand, measuring complexity (number of bytecode instructions executed by tests) would be a safer approach. – yegor256 Dec 20 '21 at 13:17
  • Ah ... so you want to measure bytecodes executed. (That's not complexity either, and I'm not sure how you do it!) But surely you will be running the build pipeline on the same hardware all of the time ... on your CI server. – Stephen C Dec 20 '21 at 13:25
  • Well, the build pipeline is first run by a programmer on a laptop and THEN on the CI server :) So, there are at least two places where the numbers should match. – yegor256 Dec 21 '21 at 05:23
  • Well unless you can find a way to get a JVM to count the byetcodes (or native instructions) executed, I'd say you should abandon any idea of getting this to work on the developers' laptops. – Stephen C Dec 21 '21 at 05:31
0

There are checkstyle Metrics for the complexity of source codes as bellow. The Cyclomatic Complexity is the tool may best fit your requirements.

  • BooleanExpressionComplexity - Restricts the number of boolean operators (&&, ||, &, | and ^) in an expression.
  • ClassDataAbstractionCoupling - Measures the number of instantiations of other classes within the given class or record.
  • ClassFanOutComplexity - Checks the number of other types a given class/record/interface/enum/annotation relies on.
  • CyclomaticComplexity - Checks cyclomatic complexity against a specified limit. It is a measure of the minimum number of possible paths through the source and therefore the number of required tests.
  • JavaNCSS - Determines complexity of methods, classes and files by counting the Non Commenting Source Statements (NCSS).
  • NPathComplexity - Checks the NPATH complexity against a specified limit. The NPATH metric computes the number of possible execution paths through a function(method).

We may enable them the maven pom.xml file

Happy
  • 757
  • 9
  • 18
  • 1
    I think that the OP is asking about Big-O complexity; i.e. the kind where an "unfortunate" code change makes the code run disastrously slow. – Stephen C Dec 20 '21 at 07:20
  • @StephenC - yes, I agree. It asked for the test case runtime complexity, and works with maven - 1, I did not find such a tool available yet, `Cyclomatic Complexity` is the most near maven plugin style tool but not the exact tool wanted by @yegor256 2, Another method was Do a JVM profiling when running the test cases, either unit test or integration test; this is the method we are using to monitor performance degradation now; well JVM profiling cannot integrate with maven but its a standalone tool unfortunately – Happy Dec 20 '21 at 07:40
  • Cyclomatic complexity of a code may be low, while at the same time its runtime one (aka performance) is bad. – yegor256 Dec 20 '21 at 09:00