4

I'm using jmh to benchmark a simple application (from SO question Unexpected Scalability results in java fork-join) using maven and following the command-line approach as advised in http://openjdk.java.net/projects/code-tools/jmh/. After successfully setting up and building the benchmark, I get the following benchmark results using the avgt mode:

C:\Users\username\my-app\test>java -jar target/benchmarks.jar -bm avgt -f 1
# JMH 1.10.1 (released 13 days ago)
# VM invoker: C:\Program Files\Java\jre1.8.0_45\bin\java.exe
# VM options: <none>
# Warmup: 20 iterations, 1 s each
# Measurement: 20 iterations, 1 s each
# Timeout: 10 min per iteration
# Threads: 1 thread, will synchronize iterations
# Benchmark mode: Average time, time/op
# Benchmark: org.sample.MyBenchmark.testMethod

# Run progress: 0,00% complete, ETA 00:00:40
# Fork: 1 of 1
# Warmup Iteration   1: ? 10?? s/op
# Warmup Iteration   2: ? 10?? s/op
# Warmup Iteration   3: ? 10?? s/op
# Warmup Iteration   4: ? 10?? s/op
# Warmup Iteration   5: ? 10?? s/op
# Warmup Iteration   6: ? 10?? s/op
# Warmup Iteration   7: ? 10?? s/op
# Warmup Iteration   8: ? 10?? s/op
# Warmup Iteration   9: ? 10?? s/op
# Warmup Iteration  10: ? 10?? s/op
# Warmup Iteration  11: ? 10?? s/op
# Warmup Iteration  12: ? 10?? s/op
# Warmup Iteration  13: ? 10?? s/op
# Warmup Iteration  14: ? 10?? s/op
# Warmup Iteration  15: ? 10?¹? s/op
# Warmup Iteration  16: ? 10?? s/op
# Warmup Iteration  17: ? 10?¹? s/op
# Warmup Iteration  18: ? 10?? s/op
# Warmup Iteration  19: ? 10?¹? s/op
# Warmup Iteration  20: ? 10?¹? s/op
Iteration   1: ? 10?¹? s/op
Iteration   2: ? 10?¹? s/op
Iteration   3: ? 10?? s/op
Iteration   4: ? 10?¹? s/op
Iteration   5: ? 10?¹? s/op
Iteration   6: ? 10?? s/op
Iteration   7: ? 10?¹? s/op
Iteration   8: ? 10?? s/op
Iteration   9: ? 10?? s/op
Iteration  10: ? 10?¹? s/op
Iteration  11: ? 10?? s/op
Iteration  12: ? 10?? s/op
Iteration  13: ? 10?¹? s/op
Iteration  14: ? 10?? s/op
Iteration  15: ? 10?? s/op
Iteration  16: ? 10?¹? s/op
Iteration  17: ? 10?? s/op
Iteration  18: ? 10?¹? s/op
Iteration  19: ? 10?¹? s/op
Iteration  20: ? 10?¹? s/op


Result "testMethod":
? 10?¹? s/op


# Run complete. Total time: 00:00:40

Benchmark               Mode  Cnt    Score     Error  Units
MyBenchmark.testMethod  avgt   20  ? 10?¹?             s/op

I'm not sure how to interpret this output, but I'm quite sure something went wrong...? Any idea what or how to debug this?

Community
  • 1
  • 1
Codiloo
  • 83
  • 4
  • 1
    If you see actual question marks instead of characters this means your output device (terminal) does not support the glyphs for the characters JMH is trying to display. It may be due to some environment variables being incorrectly set in some setups, or even an incomplete font – fge Jul 08 '15 at 12:03
  • Are you sure about this? Redirecting the output to a text file also gives me 'question marks'. When using the -rff (write to file) option, I get a summary where it says 0,000000 with unit s/o. – Codiloo Jul 08 '15 at 12:34

2 Answers2

8

JMH output makes use of extended Unicode characters. In particular, ? 10?? s/op" probably means "≈ 10⁻¹⁰ s/op". Use a terminal that supports Unicode properly, see this.

Apart from Unicode troubles, your benchmark seems to be too short, and so JMH presents an order of magnitude estimate instead of showing "0.000 s/op".

Community
  • 1
  • 1
Aleksey Shipilev
  • 18,599
  • 2
  • 67
  • 86
  • The ? in the output were indeed caused by the fact that the terminal I was using didn't support unicode properly. Looking at the MyBenchmark.java source file generated after step 1, the testMethod turned out to be empty (and therefore terminated quite quickly). I simply inserted my benchmark code in there and it worked. However, this is not the proper/best way to use jmh, right? – Codiloo Jul 09 '15 at 09:26
  • 1
    Look through JMH Samples to get the idea how to use it: http://hg.openjdk.java.net/code-tools/jmh/file/tip/jmh-samples/src/main/java/org/openjdk/jmh/samples/ – Aleksey Shipilev Jul 09 '15 at 09:35
0

To see more precise numbers in JMH output you can change the time unit to be used, e.g.

@BenchmarkMode(Mode.AverageTime)
@OutputTimeUnit(TimeUnit.MILLISECONDS)
public class MyBenchmark {
    ...
malloc4k
  • 1,742
  • 3
  • 22
  • 22