Define number of cycles - Benchmark.js

Question

I am trying to execute a sample performance benchmark using Benchmark.js. Here is what I have wrote:

var Benchmark = require('benchmark');
var arr = []
benchmark = new Benchmark('testPerf',function(){
    arr.push(1000);
},
{
    delay: 0,
    initCount: 1,
    minSamples: 1000,
    onComplete : function(){ console.log(this);},
    onCycle: function(){}
});
benchmark.run();

Now like we do in JUnitBenchmarks:

@BenchmarkOptions(clock = Clock.NANO_TIME, callgc = true, benchmarkRounds = 10, warmupRounds = 1)

Here also I want to declare benchmarkRounds and warmupRounds count in benchmarkjs. I think warmupRounds maps to initCount? And how to set exact number of cycles/benchmark iteration?

Or if we have some other good JavaScript library that can handle it would work too.

Benchmarking in JavaScript using a fixed iteration count is [problematic](http://calendar.perfplanet.com/2010/bulletproof-javascript-benchmarks/), so Benchmark.js does not allow that option. Instead, it runs tests over and over again [until results can be considered reasonably accurate](http://monsur.hossa.in/2012/12/11/benchmarkjs.html). — approxiblue, Sep 30 '15 at 01:47

score 8 · Accepted Answer · answered Oct 01 '15 at 07:21

Using fixed iteration counts in JavaScript benchmarks is risky: we might get zero-time results eventually, as browsers become faster.

Benchmark.js does not allow setting the number of rounds/iterations in advance. Instead it runs tests over and over again until results can be considered reasonably accurate. You should check out the code reading by Monsur Hossain. Some highlights from the article:

A cycle in Benchmark.js consists of set-up, tear-down, and multiple iterations of the actual test.
Benchmark.js starts with the analysis phase: run a few cycles to find the optimal number of iterations (finish the test as fast as possible while collecting enough samples to generate accurate results).
The number of cycles run during analysis is saved in Benchmark.prototype.cycles.
Knowing the optimal number of iterations, Benchmark.js begins the sampling phase: runs the tests and actually stores the results.
Benchmark.prototype.stats.sample is an array of results from each cycle during sampling.
Benchmark.prototype.count is the number of iterations during sampling.

score 5 · Answer 2 · answered Sep 30 '15 at 00:02

5

Looking over the docs at:

http://benchmarkjs.com/docs

Sounds like you are correct

warmupRounds => initCount (http://benchmarkjs.com/docs#options_initCount)
cycles => http://benchmarkjs.com/docs#prototype_cycles

answered Sep 30 '15 at 00:02

DanH

492
1
7
14

4

A cycle consists of set-up, tear-down, and multiple iterations of the test. A cycle is _not_ a single iteration of the test, as seems to be the case in [JUnitBenchmarks](http://labs.carrotsearch.com/junit-benchmarks-tutorial.html). – approxiblue Sep 30 '15 at 01:41
Cycles don't work as expected. I have defined `cycles`: 10 but function is executed 73 times. Looks like @approxiblue is correct – Tariq Sep 30 '15 at 10:12

score 3 · Answer 3 · answered May 12 '16 at 14:32

Regardless if it's a good idea, if you set minTime and maxTime to some negative values then minSamples and initCount will be left as the only criteria and they will correspond to #cycles and warmup iterations run at each cycle. So the test function will be executed (initCount+1) * minSamples times. At least that's what my experiments show.

var Benchmark = require('benchmark');
var counter = 0;
Benchmark('counting', {
  'fn': function() { ++counter; },
  minSamples: 3,
  initCount: 1,
  minTime: -Infinity,
  maxTime: -Infinity,
  onCycle: function () { console.log('[onCycle] counter: ' + counter); },
  onComplete : function(){ console.log('mean: ' + this.stats.mean);},
}).run();

gives me with benchmark.js 2.1.0:

$ node count.js
[onCycle] counter: 2
[onCycle] counter: 4
[onCycle] counter: 6
mean: 0.0000034683333333333333

Define number of cycles - Benchmark.js

3 Answers3