How to measure java performance on the i7?

Question

I'm trying to do a typical "A/B testing" like approach on two different implementations of a real-life algorithm, using the same data-set in both cases. The algorithm is deterministic in terms of execution, so I really expect the results to be repeatable.

On the Core 2 Duo, this is also the case. Using just the linux "time" command I'll get variations in execution time around 0.1% (over 10 runs).

On the i7 I will get all sorts of variations, and I can easily have 30% variations up and down from the average. I assume this is due to the various CPU optimizations that the i7 does (dynamic overclocking etc), but it really makes it hard to do this kind of testing. Is there any other way to determine which of 2 algorithms is "best", any other sensible metrics I can use ?

Edit: The algorithm does not sustain for very long and this is actually the real-life scenario I'm trying to benchmark. So running repeatedly is not really an option as such.

Is it possible to disable these CPU optimizations? Perhaps by compiling a custom kernel or setting some /proc flags? — Frank Schmitt, May 04 '11 at 08:31

score 3 · Answer 1 · answered May 04 '11 at 08:29

3

See if you can turn off the dynamic over-clocking in your BIOS. Also, ditch all possible other processes running when doing the benchmarking.

answered May 04 '11 at 08:29

Christo

8,729
2
22
16

oh, yeah, I forgot one thing: if your benchmarking doesn't use network communications, disable your NIC(s) and reboot before running the benchmarks. – Christo May 04 '11 at 08:47

score 1 · Answer 2 · answered May 04 '11 at 08:30

Well you could use O-notation principles in determining the performance of algorithms. This will determine the theoretical speed of an algorithm.

http://en.wikipedia.org/wiki/Big_O_notation

If you absolutely must know the real life speed of the alogorithm, then ofc you must benchmark it on a system. But using the O-notation you can see past all that and only focus on the factors/variables that are important.

score 0 · Answer 3 · edited May 23 '17 at 11:47

0

You didn't indicate how you're benchmarking. You might want to read this if you haven't yet: How do I write a correct micro-benchmark in Java?

If you're running a sustained test I doubt dynamic clocking is causing your variations. It should stay at the maximum turbo speed. If you're running it too long perhaps it's going down one multiplier for heat. Although I doubt that, unless you're over-clocking and are near the thermal envelope.

Hyper-Threading might be playing a role. You could disable that in your BIOS and see if it makes a difference in your numbers.

edited May 23 '17 at 11:47

Community

1
1

answered May 04 '11 at 08:37

WhiteFang34

70,765
18
106
111

This is not a microbenchmark. It's a live computation, but it doesnt sustain for too long. This is what I'm trying to benchmark – krosenvold May 04 '11 at 08:49
Perhaps the dynamic clocking is affecting you then. It sounds like you should disable that and Hyper-Threading to rule them out. Is there any reason you can't just repeat your computation (like 100x) to sustain it for longer to get a more consistent result? – WhiteFang34 May 04 '11 at 09:03

score 0 · Answer 4 · answered May 04 '11 at 08:41

On linux you can lock the CPU speed to stop clock speed variation. ;)

You need to make the benchmark as realistic as possible. For example, if you run an algorithm flat out and take an average it you might get very different results from performing the same tasks every 10 ms. i.e. I have seen 2x to 10x variation (between flat out and relatively low load), even with a locked clock speed.

How to measure java performance on the i7?

4 Answers4