I created a complete test project (version in question linked). I have based this project on several things:
- The original java repository of the same demo
Generating an archetype, e.g.:
mvn archetype:generate \ -DinteractiveMode=false \ -DarchetypeGroupId=org.openjdk.jmh \ -DarchetypeArtifactId=jmh-scala-benchmark-archetype \ -DgroupId=org.sample \ -DartifactId=test \ -Dversion=1.0One way to get the test runner working with JMH (otherwise doesn't even try to run): JMH Unable to find the resource: /META-INF/BenchmarkList
- Further confirmation in methodology: How to run JMH from inside JUnit tests?
- Gradle/idea has similar issues, maybe.
- Maybe others ... I've been staring at this for many hours, but I think that is it.
Here is the test:
package com.szatmary.peter
import org.junit.Assert
import org.junit.Test
import org.openjdk.jmh.annotations.Benchmark
import org.openjdk.jmh.annotations.Mode
import org.openjdk.jmh.annotations.Scope
import org.openjdk.jmh.annotations.State
import org.openjdk.jmh.results.BenchmarkResult
import org.openjdk.jmh.results.RunResult
import org.openjdk.jmh.runner.Runner
import org.openjdk.jmh.runner.RunnerException
import org.openjdk.jmh.runner.options.Options
import org.openjdk.jmh.runner.options.OptionsBuilder
import org.openjdk.jmh.runner.options.TimeValue
import org.openjdk.jmh.runner.options.VerboseMode
import java.util
import java.util.concurrent.TimeUnit
import com.szatmary.peter.SampleBenchmarkTest.St
import com.szatmary.peter.SampleBenchmarkTest.St.AVERAGE_EXPECTED_TIME
/**
* It is recommended to run JMH not from tests but directly from different main method code.
* Unit-tests and other IDE interferes with the measurements.
*
* If your measurements will be in second / minutes or longer than it running nechmarks from tests
* will not affect your benchmark results.
*
* If your measurements will be in miliseconds, microseconds , nanoseconds ... run your
* benchmarks rather not from tests bud from main code to have better measurements.
*/
object SampleBenchmarkTest {
@State(Scope.Benchmark) object St {
private[peter] val AVERAGE_EXPECTED_TIME = 100 // expected max 100 milliseconds
val app = new App
}
}
class SampleBenchmarkTest {
/**
* Benchmark run with Junit
*
* @throws Exception
*/
@Test
@throws[Exception]
def runTest(): Unit = {
val opt = initBench
val results = runBench(opt)
assertOutputs(results)
}
/**
* JMH benchmark
*
*/
@Benchmark
def oldWay(st: St.type): Unit = st.app.oldWay
@Benchmark
def newWay(st: St.type): Unit = st.app.newWay
/**
* Runner options that runs all benchmarks in this test class
* namely benchmark oldWay and newWay.
*
* @return
*/
private def initBench: Options = {
System.out.println("running" + classOf[SampleBenchmarkTest].getSimpleName + ".*")
return new OptionsBuilder()
.include(classOf[SampleBenchmarkTest].getSimpleName + ".*")
.mode(Mode.AverageTime)
.verbosity(VerboseMode.EXTRA)
.timeUnit(TimeUnit.MILLISECONDS)
.warmupTime(TimeValue.seconds(1))
.measurementTime(TimeValue.milliseconds(1))
.measurementIterations(2).
threads(4)
.warmupIterations(2)
.shouldFailOnError(true)
.shouldDoGC(true)
.forks(1)
.build
}
/**
*
* @param opt
* @return
* @throws RunnerException
*/
@throws[RunnerException]
private def runBench(opt: Options) = new Runner(opt).run
/**
* Assert benchmark results that are interesting for us
* Asserting test mode and average test time
*
* @param results
*/
private def assertOutputs(results: util.Collection[RunResult]) = {
import scala.collection.JavaConversions._
for (r <- results) {
import scala.collection.JavaConversions._
for (rr <- r.getBenchmarkResults) {
val mode = rr.getParams.getMode
val score = rr.getPrimaryResult.getScore
val methodName = rr.getPrimaryResult.getLabel
Assert.assertEquals("Test mode is not average mode. Method = " + methodName, Mode.AverageTime, mode)
Assert.assertTrue("Benchmark score = " + score + " is higher than " + AVERAGE_EXPECTED_TIME + " " + rr.getScoreUnit + ". Too slow performance !", score < AVERAGE_EXPECTED_TIME)
}
}
}
}
The error from running mvn clean install is:
[INFO] --- exec-maven-plugin:1.6.0:exec (run-benchmarks) @ jmh-benchmark-demo ---
Exception in thread "main" java.lang.RuntimeException: ERROR: Unable to find the resource: /META-INF/BenchmarkList
at org.openjdk.jmh.runner.AbstractResourceReader.getReaders(AbstractResourceReader.java:98)
at org.openjdk.jmh.runner.BenchmarkList.find(BenchmarkList.java:122)
at org.openjdk.jmh.runner.Runner.internalRun(Runner.java:263)
at org.openjdk.jmh.runner.Runner.run(Runner.java:209)
at org.openjdk.jmh.Main.main(Main.java:71)
[ERROR] Command execution failed.
org.apache.commons.exec.ExecuteException: Process exited with an error: 1 (Exit value: 1)
Unless requested, I won't bother including the other source file in this post (App.scala - it just has two ways of summing over an array), and it is found in the github repo.
For completeness here is my current pom file:
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
<modelVersion>4.0.0</modelVersion>
<groupId>com.szatmary.peter</groupId>
<artifactId>jmh-benchmark-demo</artifactId>
<version>1.0</version>
<name>JMH benchmark demo</name>
<prerequisites>
<maven>3.0</maven>
</prerequisites>
<properties>
<javac.target>1.8</javac.target>
<scala.version.major>2.11</scala.version.major>
<scala.version.minor>11</scala.version.minor>
<scala.version>
${scala.version.major}.${scala.version.minor}
</scala.version>
<!--
Select a JMH benchmark generator to use. Available options:
default: whatever JMH chooses by default;
asm: parse bytecode with ASM;
reflection: load classes and use Reflection over them;
-->
<jmh.generator>default</jmh.generator>
<!--
Name of the benchmark Uber-JAR to generate.
-->
<uberjar.name>benchmarks</uberjar.name>
<project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
<jmh.version>1.20</jmh.version>
<java.version>1.8</java.version>
<junit.version>4.12</junit.version>
</properties>
<dependencies>
<dependency>
<groupId>org.scala-lang</groupId>
<artifactId>scala-library</artifactId>
<version>${scala.version}</version>
</dependency>
<dependency>
<groupId>org.openjdk.jmh</groupId>
<artifactId>jmh-core</artifactId>
<version>${jmh.version}</version>
<scope>test</scope>
</dependency>
<dependency>
<groupId>org.openjdk.jmh</groupId>
<artifactId>jmh-generator-annprocess</artifactId>
<version>${jmh.version}</version>
<scope>test</scope>
</dependency>
<dependency>
<groupId>junit</groupId>
<artifactId>junit</artifactId>
<version>${junit.version}</version>
<scope>test</scope>
</dependency>
</dependencies>
<build>
<plugins>
<!--
1. Add source directories for both scalac and javac.
-->
<plugin>
<groupId>org.codehaus.mojo</groupId>
<artifactId>build-helper-maven-plugin</artifactId>
<version>1.8</version>
<executions>
<execution>
<id>add-source</id>
<phase>generate-sources</phase>
<goals>
<goal>add-source</goal>
</goals>
<configuration>
<sources>
<source>${project.basedir}/src/main/scala</source>
<source>${project.basedir}/target/generated-sources/jmh</source>
</sources>
</configuration>
</execution>
<execution>
<id>add-test-source</id>
<phase>generate-test-sources</phase>
<goals>
<goal>add-test-source</goal>
</goals>
<configuration>
<sources>
<source>${project.basedir}/src/test/scala</source>
<source>${project.basedir}/target/generated-test-sources/jmh</source>
</sources>
</configuration>
</execution>
</executions>
</plugin>
<!--
2. Compile Scala sources
-->
<plugin>
<groupId>net.alchim31.maven</groupId>
<artifactId>scala-maven-plugin</artifactId>
<version>3.3.1</version>
<configuration>
<recompileMode>incremental</recompileMode>
</configuration>
<executions>
<execution>
<phase>process-sources</phase>
<goals>
<goal>compile</goal>
</goals>
</execution>
</executions>
</plugin>
<!--
4. Compile JMH generated code.
-->
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-compiler-plugin</artifactId>
<version>3.1</version>
<configuration>
<compilerVersion>${javac.target}</compilerVersion>
<source>${javac.target}</source>
<target>${javac.target}</target>
<compilerArgument>-proc:none</compilerArgument>
</configuration>
</plugin>
<plugin>
<groupId>org.codehaus.mojo</groupId>
<artifactId>exec-maven-plugin</artifactId>
<executions>
<execution>
<id>run-benchmarks</id>
<phase>integration-test</phase>
<goals>
<goal>exec</goal>
</goals>
<configuration>
<classpathScope>test</classpathScope>
<executable>java</executable>
<arguments>
<argument>-classpath</argument>
<classpath />
<argument>org.openjdk.jmh.Main</argument>
<argument>.*</argument>
</arguments>
</configuration>
</execution>
</executions>
</plugin>
</plugins>
</build>
</project>
