EventEmitter vs RxJS vs Kefir

Question

I wanted to compare JS EventEmitter and RxJS performance. I wrote following benchmark script to do that:

Performance Test

import Rx from 'rxjs/Rx';
import Kefir from 'kefir';

import { EventEmitter } from "events";

let Benchmark = require ("benchmark");
let suite = new Benchmark.Suite;

suite
.add('for', () => {
  let numArray = [1,2,3,4,5,6,7,8,9,10];
  let count = 0;
  for (let i = 0; i<numArray.length; i++)
    count += numArray[i];
})
.add('forEach', () => {
  let numArray = [1,2,3,4,5,6,7,8,9,10];
  let count = 0;
  numArray.forEach((num) => { count += num; });
})
.add('eventEmitter', () => {
  let numArray = [1,2,3,4,5,6,7,8,9,10];
  let count = 0;
  let myEmitter = new EventEmitter();
  myEmitter.on('number', (num) => { count += num; });
  numArray.forEach((num) => { myEmitter.emit('number', num); });
})
.add('rxjs', () => {
  let numArray = [1,2,3,4,5,6,7,8,9,10];
  let count = 0;
  let source = Rx.Observable.from(numArray)
    .do((x) => { count += x }, (error) => {}, () => {});
  source.subscribe((x) => {}, (error) => {}, () => {});
})
.add('kefir', () => {
  let numArray = [1,2,3,4,5,6,7,8,9,10];
  let count = 0;
  let stream = Kefir.sequentially(0, numArray);
  count = stream.scan(sum => sum + 1, 0);
})
.on('cycle', function (event) {
  console.log(String(event.target));
})
.on('complete', function () {
  console.log('Slowest is ' + this.filter('slowest').map('name'));
})
.run({'async': true});

Performance Results

for x 47,595,916 ops/sec ±1.58% (87 runs sampled)
forEach x 4,428,485 ops/sec ±0.75% (86 runs sampled)
eventEmitter x 1,478,876 ops/sec ±0.61% (86 runs sampled)
rxjs x 547,732 ops/sec ±0.66% (86 runs sampled)
kefir x 496,709 ops/sec ±5.15% (50 runs sampled)
Slowest is kefir

As you can see Kefir turned out to be slowest contrary to the claim made at this link.

1. Have I done something wrong in writing the test?
2. It would be great if anyone can explain the difference why does it occur. Specially when you compare it with javascript event-emitter.

Answer 1

I know you asked this question a while ago, but I think it might be helpful to give future readers an idea of some of the problems I noticed while looking at the benchmarks.

Firstly, count isn't a number in Kefir, it's a stream. Try logging count after you call scan. And importantly, it seems the Kefir count stream is never activated and the computation is never run! That needs to be fixed first. I suspect the same is true for the Rx benchmark, but you'll have to check the documentation to see if creating a stream from an ES observable will block or not.

I believe you can implement an async test with that library with something similar to (untested code):

suite.add('My async test', function(deferred) {
    let numArray = [1,2,3,4,5,6,7,8,9,10];
    let stream = Kefir.sequentially(0, numArray);
    let countStream = stream.scan(sum => sum + 1, 0);
    countStream.onEnd(() => deferred.resolve());
}, {'defer': true})

With that in mind, it's very strange Kefir's benchmark is the slowest because it does NO WORK. I suspect the test arrays are too small and are calculated too quickly to get a valuable benchmark. It's likely tests are actually measuring the stream construction times or whichever benchmark happens to have the most garbage collected/processes by the runtime while it's running. In fact, if the Rx benchmark doesn't wait for the test to finish, it will be doing it's processing/garbage cleanup DURRING the Kefir test! You can mitigate this by waiting for the Kefir and Rx tests to finish in the benchmark; create less garbage by reusing a common, global test array between benchmarks; and use a very, very large array to be sure iteration is what dominates the time spent in the benchmarks.

Finally, for the asynchronous benchmarks (Kefir and Rx), you need to be sure the benchmarks are processing events the same way with respect to the event loop. I'm certain the Kefir example processes each event in a DIFFERENT tick of the event loop and will have to wait for any of the browser's other activity (rendering/painting, other callbacks/timeouts, etc.) to complete between each step in the stream. Consider the output of the following code:

console.log('a')
setTimeout(function() {
    console.log('b')
}, 0);
console.log('c')

This code will always print a, c, b with a minor, nonzero delay in printing the final b.

I believe you can get Kefir to process the array in the same tick with something like Kefir.constant([1,2,3...10]).flatten(). However, I don't think it's very useful to compare the two streaming frameworks in a synchronous benchmark because that is not their intended purposes.

Finally, the scan operation is semantically different than a forEach / do in the other frameworks because it produces a value for the output stream at each step for any potential listeners while the others only have to run that one block of code.

Benchmarks are very hard to get right, but I hope this helps.