How can I make transitions between dc.js charts smooth when I have a map with more than 20k points?

Question

I've recently built a smaller version of a prototype data explorer incorporating crossfilter, dc.js, and leaflet.markerCluster. The small version, (prototype dashboard), works properly. The problem I am having is when I try to scale it up to 20k points or more.

The charts still render correctly, and the map works to update the charts smoothly when zooming or panning, but when I interact with one of the charts, the transitions between the other charts are no longer smooth. They jump to their next position rather than smoothly transitioning.

I tried removing the map and this restored the transitions between the other charts to a nice smooth transition again.

I'm wondering if the re-rendering process is getting caught up with the 20k points each time an interaction occurs.

If anyone has any suggestions about where I might look for a solution I'd be grateful.

Answer 1

Thanks for posting a block, that makes things easier to test.

I simulated a lot more points by generating 200 rows for each of yours ~ 46k rows. I saw only a little stuttering at 100x ~ 23k rows (2017 iMac with plenty of RAM).

Leaflet.markercluster is known to be slow with more than 10K points. With 46k rows it took about 475ms for Leaflet.markercluster to clear and add the Leaflet layers:

Since there is only one thread in JavaScript (unless you use workers), D3 needs to get timeouts (actually requestAnimationFrame) every 16ms or so in order to produce fluid animation.

One workaround is to delay the map redrawing 500ms until the others have done:

  dc.override(mapChart, 'redraw', function() {
      window.setTimeout(() => mapChart._redraw(), 500);
  });

Fork of your block with workaround.

Of course, this also makes the map take 500ms longer to redraw. And if you click around fast enough, the last map redraw will still be running when it's trying to draw the charts.

You could also try the chunked addLayers options but I think you would have to set the chunkedInterval so low that it would also slow down the markerclusters.

Processing this much data efficiently is possible in JavaScript - obviously crossfilter has no problem here. I don’t know if the cloistering algorithm is inherently too expensive. Someone on the issue suggested pre-aggregating the points, but I think this would mean you wouldn't be able to see individual points.