Simple performance test, different results

Question

I am made performance test of collection, but every time, when i am starting test, i have different results. Why it is happening and what can i do for take correct results? I am think my problem with option VM.

public class PerformanceTest {
private static void addToTheBeginTest() {
    String nameOfMethod = "Method add 250_000 result: ";

    List<Integer> arrayList = new ArrayList<>();
    StopWatch.start();
    for (int i = 0; i < 250_000; i++) {
        arrayList.add(0, 1);
    }
    long resultAL = StopWatch.getElapsedTime();

    outputInFile(nameOfMethod, resultAL);
}

private static void outputInFile(String nameOfMethod, long resultAl) {
    File outputFile = new File("D:\\Idea Project\\ExperimentalProject\\src\\SimplePerformance\\");
    outputFile.mkdir();
    try (FileWriter writer = new FileWriter("D:\\Idea Project\\ExperimentalProject\\src\\SimplePerformance\\SimplePerformanceTest.txt", true)) {
        writer.write(nameOfMethod);
        writer.write(String.valueOf(resultAl) + " mc \n");
    } catch (IOException e) {
        System.out.println(e.getMessage());
    }
}

class StopWatch {
private static long result;

public static void start() {
    result = System.currentTimeMillis();
}

public static long getElapsedTime() {
    return System.currentTimeMillis() - result;
   }
}

Results of 3 times

you wont have the time equal on a ms scale if you execute the same code twice after each other. But the results are pretty damn close to each other, and the average of them is rather your execution time. — SomeJavaGuy, May 23 '16 at 14:09
@meskobalazs, different results, iam expect always the same time. — diofloyk, May 23 '16 at 14:11
Obligatory link: http://stackoverflow.com/questions/504103/how-do-i-write-a-correct-micro-benchmark-in-java — Andy Turner, May 23 '16 at 14:13
@KevinEsche, thx, but why? and is it possible to get the same amount of time? Sorry for my English :) — diofloyk, May 23 '16 at 14:14
You shouldn't expect the exact same results every time. Even though your code is doing the same every time, there will be other things happening on your computer; some of them will take time, some of them will change what is and what isn't in your cache, etc. And each time you run your program, it will (again, mostly for cache-related reasons) slightly change the environment for the next time you run. — Gareth McCaughan, May 23 '16 at 14:14
The execution time won't always be the same, even on the same machine and the same CPU/RAM/HDD usage. — agilob, May 23 '16 at 14:14

score 3 · Accepted Answer · answered May 23 '16 at 14:14

3

The reasons for this have already been explained by other answers.

The easiest way to handle this is to run each test x1000 and take an average. You should find the results more consistent.

answered May 23 '16 at 14:14

Lee

738
3
13

2

It's 1000 because if you look at OP a single run of the test takes 7 seconds. Unless OP wants to wait 4.6 weeks (as your 400000 would require) for the results of his test then he would do well to choose a reasonable number of iterations to average. – Lee May 23 '16 at 14:31

score 2 · Answer 2 · answered May 23 '16 at 14:18

2

The exact amount of resources available before running a code keeps on varying every ms. It is not possible to keep it constant on regular computers.

answered May 23 '16 at 14:18

VIP

35
6

1

Why don't you come up with something constructive and save everyone's time? – VIP May 23 '16 at 14:33
Do you call this constructive? `The exact amount of resources available before running a code keeps on varying every ms. It is not possible to keep it constant on regular computers.` This comment is shorter and more informative than yours : `The execution time won't always be the same, even on the same machine and the same CPU/RAM/HDD usage` – UDKOX May 23 '16 at 14:36
You didnt answer my question. Are you here just to read comments and compare? – VIP May 23 '16 at 14:42
I am here to explain to new users that answers should like more like [this](http://stackoverflow.com/questions/37352117/how-can-i-make-this-program-faster-and-simple/37365773#37365773) or [this](http://stackoverflow.com/questions/37380389/how-to-make-a-frame-display-50-times-for-1-second-and-disappear-each-time-its-d/37380800#37380800), and not like: `The exact amount of resources available before running a code keeps on varying every ms. It is not possible to keep it constant on regular computers.` I know you cannot add comments yet, but try to write more in depth answers. – UDKOX May 23 '16 at 14:48

score 1 · Answer 3 · answered May 23 '16 at 14:11

1

jvms do optimisations under the hood the more time a particular code is run. You should look into micro bench marking. There's a whole bunch of things that need to be done in order to get accurate results.

answered May 23 '16 at 14:11

mattsap

3,790
1
15
36

Simple performance test, different results

3 Answers3