7

As you can see in code below, when I increase the size of the string it leads to a 0 milliseconds difference. And moreover there is an inconsistency as the string count goes on increasing.

Am I doing something wrong here?

let stringIn = document.getElementById('str');
let button = document.querySelector('button');

button.addEventListener('click', () => {
  let t1 = performance.now();
  functionToTest(stringIn.value);
  let t2 = performance.now();
  console.log(`time taken is ${t2 - t1}`);
});

function functionToTest(str) {
  let total = 0;
  for(i of str) {
   total ++;
  }
  return total;
}
<input id="str">
<button type="button">Test string</button>

I tried using await too, but result is the same (see code snippet below). The function enclosing the code below is async:

let stringArr = this.inputString.split(' ');
let longest = '';
const t1 = performance.now();
let length = await new Promise(resolve => {
  stringArr.map((item, i) => {
    longest = longest.length < item.length ? longest : item;
    i === stringArr.length - 1 ? resolve(longest) : '';
  });
});
const diff = performance.now() - t1;
console.log(diff);
this.result = `The time taken in mili seconds is ${diff}`;

I've also tried this answer as, but it is also inconsistent.

As a workaround I tried using console.time feature, but It doesn't allow the time to be rendered and isn't accurate as well.


Update: I want to build an interface like jsPerf, which will be quite similar to it but for a different purpose. Mostly I would like to compare different functions which will depend on user inputs.

Black Mamba
  • 13,632
  • 6
  • 82
  • 105
  • 6
    Yes, Meltdown and Spectre killed `performance.now()`... Depending on the UA, it is now either rounded (to 1ms in FF, higher precision in others) or jittered, or both... I can't tell for every UA, but you can actually leverage for yourself these restrictions in FF from about:config > privacy.reduceTimerPrecision & privacy.resistFingerprinting.reduceTimerPrecision.jitter – Kaiido Aug 29 '18 at 05:23
  • Ohh but I can't build an app with that which will be consistent across the browsers. @Kaiido. Any other option should I go with console.time now ? – Black Mamba Aug 29 '18 at 05:25
  • 2
    run the test with a few thousand strings, measure that, then divide the time. – Jonas Wilms Aug 29 '18 at 05:29
  • 3
    ...Not sure... But anyway, since what you are doing is testing different execution times, you have to know that even with full precision perf.now, running a single instance of each function is useless. You must run batches of these functions to avoid potential glitches. And if you run batches big enough, the ms limitation might not be that much of a problem to detect which code runs better. – Kaiido Aug 29 '18 at 05:29
  • I'm processing the functions based on user inputs so I think I'll need an algorithm to multiply the string count so that I can get a good performance measure. – Black Mamba Aug 29 '18 at 05:35
  • basically you asking why does it stay at 0 after hitting the button 3 times. what happens if you swop your 'let's for var – Seabizkit Aug 29 '18 at 06:58
  • Yes a lot of confusion but I just want an accurate method to test the function run time. – Black Mamba Aug 29 '18 at 07:26
  • What are you trying to achieve? Measuring this *specific* function (that counts letters), or is this an example to explain your problem? Why are you interested in user input dependant performance? – Amit Sep 11 '18 at 06:25
  • Updated my question for that @Amit Can you please help a little bit as I can't force the user to enter a long string and Adding my own data to string will change the time taken. – Black Mamba Sep 11 '18 at 06:41
  • 4
    Benchmarking code is very complicated and JavaScript (as any other JIT) only makes that harder. I suggest you read about benchmark biases to get some ideas but you'll have to overcome warm-up effects, late runtime optimizations and external interferences. Good luck ;-) – Amit Sep 11 '18 at 16:11
  • Already a lot to do thanks bhai :-) @Amit – Black Mamba Sep 11 '18 at 16:36
  • https://mrale.ph/blog/2012/12/15/microbenchmarks-fairy-tale.html – Bergi Sep 15 '18 at 18:46

3 Answers3

8

There are 3 things which may help you understand what happening:

  1. Browsers are reducing performance.now() precision to prevent Meltdown and Spectre attacks, so Chrome gives max 0.1 ms precision, FF 1ms, etc. This makes impossible to measure small timeframes. If function is extremely quick - 0 ms is understandable result. (Thanks @kaiido) Source: paper, additional links here

  2. Any code(including JS) in multithreaded environment will not execute with constant performance (at least due to OS threads switching). So getting consistent values for several single runs is unreachable goal. To get some precise number - function should be executed multiple times, and average value is taken. This will even work with low precision performance.now(). (Boring explanation: if function is much faster than 0.1 ms, and browser often gives 0ms result, but time from time some function run will win a lottery and browser will return 0.1ms... longer functions will win this lottery more often)

  3. There is "optimizing compiler" in most JS engines. It optimizes often used functions. Optimization is expensive, so JS engines optimize only often used functions. This explains performance increase after several runs. At first function is executed in slowest way. After several executions, it is optimized, and performance increases. (should add warmup runs?)


I was able to get non-zero numbers in your code snipet - by copy-pasting 70kb file into input. After 3rd run function was optimized, but even after this - performance is not constant

time taken is 11.49999990593642
time taken is 5.100000067614019
time taken is 2.3999999975785613
time taken is 2.199999988079071
time taken is 2.199999988079071
time taken is 2.099999925121665
time taken is 2.3999999975785613
time taken is 1.7999999690800905
time taken is 1.3000000035390258
time taken is 2.099999925121665
time taken is 1.9000000320374966
time taken is 2.300000051036477

Explanation of 1st point

Lets say two events happened, and the goal is to find time between them. 1st event happened at time A and second event happened at time B. Browser is rounding precise values A and B and returns them.

Several cases to look at:

A       B       A-B         floor(A)    floor(B)    Ar-Br      
12.001  12.003  0.002       12          12          0
11.999  12.001  0.002       11          12          1
Andrii Muzalevskyi
  • 3,261
  • 16
  • 20
2

Browsers are smarter than we think, there are lot's of improvements and caching techniques take in place for memory allocation, repeatable code execution, on-demand CPU allocation and so on. For instance V8, the JavaScript engine that powers Chrome and Node.js caches code execution cycles and results. Furthermore, your results may get affected by the resources that your browser utilizes upon code execution, so your results even with multiple executions cycles may vary.

As you have mentioned that you are trying to create a jsPerf clone take a look at Benchmark.js, this library is used by the jsPerf development team.

Running performance analysis tests is really hard and I would suggest running them on a Node.js environment with predefined and preallocated resources in order to obtain your results.

vorillaz
  • 6,098
  • 2
  • 30
  • 46
  • What do you mean by "*execution cycles and results*"? They're not mentioned in the article you linked. – Bergi Sep 15 '18 at 18:46
  • I will keep this in mind thanks though. Using node.js adds one huge extra step in my development for which I don't have time now. I think I can look forward to using benchmark.js – Black Mamba Sep 15 '18 at 19:00
0

You might what to take a look at https://github.com/anywhichway/benchtest which just reuses Mocha unit tests. Note, performance testing at the unit level should only be one part of your performance testing, you should also use simulators that emulate real world conditions and test your code at the application level to assess network impacts, module interactions, etc.

AnyWhichWay
  • 716
  • 8
  • 11