How to benchmark host multi-threaded CPU performance in Java?

Question

I need to create a simple Java app that returns just one number: estimated CPU performance. For example when I run it on machine with 4 cores I will roughly get twice as big number than if run with 2 cores. This app should use 100% CPU for several seconds to measure that. I'm really not worried about accuracy.

I was really surprised that I couldn't find any Java library that already does that. Of course there are tools in other languages, but in my environment only Java is approved.

My current idea is to use classes from SciMark 2.0 in my code and run it from multiple threads, however this tool looks very messy (e.g. class names beginning with lowercase letters) and I need to write custom code to run these threads and combine the results.

Can I do any better to solve that problem?

Answer 1

This is simplest piece of code that does what I wanted. It tries to estimate CPU performance for multiple threads by calculating sum of square roots for subsequent integers. Variable iterations could be adjusted to increase/decrease length of benchmark. On my machine with default values it takes about 7 seconds.

import static java.util.stream.IntStream.rangeClosed;

class Benchmark {
    public static void main(String[] args) {
        final int iterations = 100_000_000;
        long start = System.currentTimeMillis();
        rangeClosed(1, 50).parallel()
                .forEach(i -> rangeClosed(1, iterations).mapToDouble(Math::sqrt).sum());
        System.out.println(System.currentTimeMillis() - start);
    }
}

Answer 2

If I understand you correctly, your goal is to measure system performance rather than application performance.

Here's the problem. System performance cannot be reduced to a single meaningful number. In reality, system performance ... even CPU performance is multi-dimensional.

For example, an application that memory intensive will perform differently on different machines depending on the CPU chip's memory cache size and design ... and the memory speed. But if the application is compute intensive, then the performance will depend more on the clock rate and core count.

Then there are issues like the effects of NUMA cells and thread pinning when the core count is high and/or you have multiple CPU chips.

These and similar issues are why benchmarks that attempt to measure raw CPU performance independent of the application have largely fallen out of favor. (MIPS originally meant million (hardware) instructions per second. It is now often referred as mythical instructions per second ... alluding to the bogosity of the measure as a predictor of real application performance)

Answer 3

Java Mcrobenchmark Harness (JMH) is a toolkit to implement benchmarks of Java code.

It measures Throughput or Average Time; you could use that to estimate cpu cycles.

Basically, you need to annotate with @Benchmark the method you want to benchmark. This method

Thare are few JMH usage samples in their repository.

It is always recommended to let the computer alone while it runs the benchmarks, and you should close all other applications (if possible). If your computer is running other applications, these applications may take time from the CPU and give incorrect (lower) performance numbers.

If you want to dig further in CPU performance (cycles, cache usages, instructions, etc) you will probably need to use Linux perf

Answer 4

Michal, thanks for your answer, I borrowed and added some threading to help me diagnose a virtual CPU performance issue on a client's AIX machine.

import static java.util.stream.IntStream.rangeClosed;

public class Main {

    public static void main(String[] args) {
        if (args.length < 2) {
            System.out.println("Usage: benchmark [million iterations] [maxThreads]");
            return;
        }

        final int MILLION = 1_000_000;
        final int iterations = Integer.parseInt(args[0]);
        final int maxThreads = Integer.parseInt(args[1]);

        for (int threads = 1; threads < maxThreads; threads++) {
            long start = System.currentTimeMillis();
            int count = iterations * MILLION / threads;
            rangeClosed(1, threads).parallel()
                .forEach(i -> rangeClosed(1, count).mapToDouble(Math::sqrt).sum());

            System.out.println(String.format("Benchmark of %d M iterations on %d thread(s): %d ms", iterations, threads, System.currentTimeMillis() - start));
        }

    }

}