Why is the purpose for setup, teardown, and cycles in Benchmark.js?

Question

I know the formal descriptions:

Setup: create the expected state for the test.
Teardown: Do necessary clean-up operations.

However, why is this necessary, especially in Benchmark.js ? Why the need for different test cycles (as defined in Benchmark.js in this article) ? I've observed that in all cases I can think of (and I think in all other cases as well), you can just move the set-up code to the preparation code (code outside of the benchmark/test), and perhaps the teardown code to the end of your code, and the functionality is essentially the same (I've looked at a few jsperf.com tests as well, and as far as I know, this is true of them as well).

For example, here is a benchmark I created, this version uses Setups and Teardowns:

const bench = new Benchmark(
  'TicTacToeBenchmark',
  // The function to test
  () => {
    ticTacToe.addEvent(
      'turn',
      player => {
        turnText.innerHTML =
          'It\'s ' + (player['id'] === 1 ? 'X' : 'O') + '\'s turn.';
      }
    );
  },
  {
    'setup': () => {
      const players = [
        {
          char: '&#10005;',
          className: 'playerX',
          id: 1,
        },
        {
          char: '&#9711;',
          className: 'playerY',
          id: 2,
        },
      ];
      const ticTacToe = new TicTacToe(3, players);
    }
  }
);

bench.run();

console.log(bench); // 'mean' is 5e-7 seconds

Same example, except, everything necessary for the test is declared with the rest of the page:

const players = [
  {
    char: '&#10005;',
    className: 'playerX',
    id: 1,
  },
  {
    char: '&#9711;',
    className: 'playerY',
    id: 2,
  },
];
const ticTacToe = new TicTacToe(3, players);

const bench = new Benchmark(
  'TicTacToeBenchmark',
  // The function to test
  () => {
    ticTacToe.addEvent(
      'turn',
      player => {
        turnText.innerHTML =
          'It\'s ' + (player['id'] === 1 ? 'X' : 'O') + '\'s turn.';
      }
    );
  }
);

bench.run();

console.log(bench); // 'mean' is 7e-7 seconds

Mayhaps the difference is more discernible in Unit Testing ? I don't know. Could you please provide a few cases where this would be different ? Or explain why the tests have to be run within iterations which are run within cycles (essentially, like 2 loops, cycles being the outer one) ?

All content that I can find online on this topic is basically a re-utterance of the definitions of Setup and Teardown with different wording, and unfortunately, there is no Wikipedia entry for this topic.

A large difference (that can affect performance) is the scope of `players` and `ticTacToe` — Bergi, May 22 '18 at 15:35
Oh, it shows in your example, yes! Accessing global variable is slower than accessing local scope variables (because the latter can be inlined much easier by the compiler). — Bergi, May 22 '18 at 15:39
@Bergi But is this the only difference ? Is the practical difference this makes large enough to justify the complexity introduced by setups/teardowns (I am certain it is not) ? — doubleOrt, May 22 '18 at 15:40
In many situations, setup and teardown can be very expensive, such as filling in a large array with random data that you want to test with. This time could easily swamp the time used for benchmarking the time that the code you care about uses. — Barmar, May 22 '18 at 15:41
@Barmar But you could just move that to the preparation code and there would be no difference. — doubleOrt, May 22 '18 at 15:44
Not sure what you mean. There is no `preparation` option in benchmark.js. Setup *is* preparation. If you mean something you do before calling preparation.js, the point is to provide all the information needed to reproduce a benchmark in one set of parameters. — Barmar, May 22 '18 at 15:48
@doubleOrt Yes, as you can see in your example the difference is two orders of magnitude :-) Not sure whether that's the only difference. I think `setup` also runs more often, which means you get to better randomise your bench data. — Bergi, May 22 '18 at 15:48
@Barmar I am not sure what you mean by ` something you do before calling preparation.js`, I basically mean the rest of the page (as in my second example), this is what it's called in jsperf.com — doubleOrt, May 22 '18 at 15:55
@Barmar While this does justify a setup step to some extent, I am not sure if it justifies teardowns and running the code for multiple cycles. — doubleOrt, May 22 '18 at 15:58
jsperf distinguishes between the HTML page, which is in preparation and generally loads libraries, and Javascript setup code. They're really both part of setup. Yes, you could put ` — Barmar, May 22 '18 at 15:58
@Barmar I still don't get how this makes it easier to organize, wouldn't it be much simpler (with the exception of `the point is to provide all the information needed to reproduce a benchmark in one set of parameters` as you mentioned) to not have these steps and instead declare everything in one place (HTML and setup JS) and then add the code for the tests in another place ? — doubleOrt, May 22 '18 at 16:01
@Bergi `I think setup also runs more often, which means you get to better randomise your bench data`, it does, but could you elaborate on what you mean by `better randomise your bench data` ? — doubleOrt, May 22 '18 at 16:08
@doubleOrt Simply that every run gets its own, new generated data. This should give better results with more consistency. — Bergi, May 22 '18 at 17:24
@Bergi https://stackoverflow.com/questions/8389639/unittest-setup-teardown-for-several-tests This question says that setups and teardowns run for each test case, I think it would be more justified that way, e.g if I wanted to test adding an element and then retrieving it from the DOM I would empty it in the teardown after each test, the problem is that benchmark.js functionality isn't like this, it makes you wonder why it's different. — doubleOrt, May 22 '18 at 17:35
@doubleOrt Afaik, benchmark.js runs the code under review in a timed loop between setup and teardown. Then that whole thing is ran multiple times to get an average. — Bergi, May 22 '18 at 17:39
@Bergi Yep, but the "whole thing" is run for like 50 times, and the code to be tested is run thousands and thousands of time, I now think the "50 times" are mostly there for something statistics-related. Here is a relevant issue: https://github.com/bestiejs/benchmark.js/issues/132 — doubleOrt, May 22 '18 at 17:41
I think I will have a better change for an answer if I just ask the question there. — doubleOrt, May 22 '18 at 17:43

score 1 · Answer 1 · answered Apr 29 '20 at 06:43

Setup and teardown is where you put code that (1) needs to run before or after the function you want to benchmark but (2) don't want to include in the benchmark's measurement.

For example, let's say you have a text search library. Usage is as follows:

searchEngine = new New SearchEngine(pathToLargeCorpusOfText)
searchEngine.search(queryString)
searchEngine.close() - deallocates memory

If you want to benchmark the search() method in isolation, i.e. without (1) and (3) affecting the results, you put them in setup and teardown, respectively.

To accurately benchmark a function, it must be run many times. Benchmark.js runs many iterations (one call of search()) each cycle, and runs many cycles for each benchmark. I can't explain the reasons any better than the authors of Benchmark.js: Bulletproof JavaScript benchmarks by Mathias Bynens and John-David Dalton.

Why is the purpose for setup, teardown, and cycles in Benchmark.js?

1 Answers1