Generating heatmap layer for milions of points

Question

I'm using heatmap layer from Google Maps to display a heatmap, however, I now have too many points and it stopped working, because browser cannot handle it anymore. I've found that they provide Fusion Tables, but they're also limited: to 100k rows, which is way too low. I need to render heatmap of milions or maybe even more points. I'd be perfect if my server can have some PHP script to render a heatmap, for example, once a day. And then a client from js will just download a preloaded heatmap (on the map like google maps, but may be different map too). Is this possible with some existing technology (can be commercial)?

Do you really need to show ALL points at ALL times? Could you not load only the points that fall within the viewport, maybe limit the minimum zoom level, the map container dimensions, etc.? — MrUpsidown, May 17 '18 at 15:04
Yes and no. For now, all points are around one city and I want to see the whole city (about 1.5 mil points as for today). Later on, it'll be available for other cities so then I can think of it, but as I said it's still too much for one city. I need prerendered heatmap :) — Makalele, May 18 '18 at 06:20
Mapbox pretends you can load millions of points on a map. Have a look at their blog. I found [this post](https://blog.mapbox.com/heatmaps-at-scale-for-business-intelligence-285dbbc3d7b3) and [this post](https://blog.mapbox.com/clustering-millions-of-points-on-a-map-with-supercluster-272046ec5c97). Might be worth trying. — MrUpsidown, May 18 '18 at 08:00
You can add an image overlay. Here is a link to the google maps developer documentation: https://developers.google.com/maps/documentation/javascript/customoverlays#add Pre-generate heat maps and add to the javascript as an overlay. — Jeff Harris, May 18 '18 at 18:37
I've tried MapBox: https://www.mapbox.com/mapbox-gl-js/example/heatmap-layer/ but it loads points from json file. Well it could barely work with 100 000 points after really long loading. — Makalele, May 21 '18 at 12:20
[Geo heat map(google) with large data](https://stackoverflow.com/questions/39432823/geo-heat-mapgoogle-with-large-data) this discussion should help! — Guest0x20, May 22 '18 at 11:58
For fast heatmap generation of million points use something like [WebGL Heatmap](https://github.com/pyalot/webgl-heatmap) which accomplishes high-performance by harness power of video-card through GL/pixel shaders — Agnius Vasiliauskas, May 23 '18 at 08:45
Consider looking at caching tiles of data on something like [geoserver](http://geoserver.org/) that way you can have a long running task update the layer on the server and still maintain an interactive map which uses that layer. — apokryfos, May 23 '18 at 12:11

score 10 · Accepted Answer · answered May 21 '18 at 20:36

All you need is to pre-clusterize your points into smaller number of points and pass these groups to Google Maps as if it were your original points. So, if you had 3 nearby points with values 3, 5 and 10, you create one point with value 18 at weighted average of their coordinates. Do it for all your dataset and you will reduce it 3 times. Change 3 to any appropriate value to further reduce your dataset.

The easy way to cluster your data is to use geohashing[1]. Here is nice geohashing library for PHP[2]. You may calculate a set of geohashes of different precision for each of your points only once when adding, and then use desired precision to reduce your dataset using simple GROUP BY. Example (meta):

use Lvht\GeoHash;

class DataPoint extends ActiveRecord {
    public geo_hash_precision4, geo_hash_precision5, geo_hash_precision6;
    public function save() {
        $this->geo_hash_precision4 = GeoHash::encode($this->lat,$this->lon, 0.0001);
        $this->geo_hash_precision5 = GeoHash::encode($this->lat,$this->lon, 0.00001);
        $this->geo_hash_precision6 = GeoHash::encode($this->lat,$this->lon, 0.000001);
        parent::save();
    }
}

class DataSet extends ActiveQuery {
    /**
     * @param int $p Desired precision
     */
    public function getValues($p = 6) {
        $data = $this->rawQuery("SELECT geo_hash_precision{$p}, SUM(value) FROM {$this->table} GROUP BY geo_hash_precision{$p}");
        // Add bounding box WHERE to reduce set for map size
        foreach ($data as $row) {
            list($minLon, $maxLon, $minLat, $maxLat) = GeoHash::decode($row["geo_hash_precision{$p}"]);
            $row['lat'] = ($maxLat - $minLat) / 2;
            $row['lon'] = ($maxLon - $minLon) / 2;
            unset($row["geo_hash_precision{$p}"]);
        }
    }
}

I don't have a possibility to test this answer right now, but it looks like it's the best solution for my problem :) — Makalele, May 24 '18 at 10:25
Please be advised that this is kinda pseudocode to get you an idea, real implementation will vary according to your db, ORM and framework. Feel free to ask additional questions. — SergeAx, May 25 '18 at 20:05

score 3 · Answer 2 · answered May 18 '18 at 08:34

3

I use heatmap.js for visualization. It's really fast and can handle alot of points. Not sure though if 1.5 mil points will work.

What I can imagine is to use one of the Javascript solutions with node.js to prerender. A quick google search about 'heatmap js nodejs prerender' found me this https://github.com/substack/node-heatmap and this https://mango-is.com/blog/engineering/pre-render-d3-js-charts-at-server-side/

answered May 18 '18 at 08:34

Danmoreng

2,367
1
19
32

It can be tens of milions of points in the future. Can it be run on php server on cron task? – Makalele May 18 '18 at 11:11
Well nodejs can be installed on a linux server and surely run via cron task. The only PHP heatmap server-side rendering I found was fusion-tables, which you already mentioned does not meet your requirement. So setting up a small nodejs script and try generating a heatmap with that might be your best option. – Danmoreng May 18 '18 at 14:03

Generating heatmap layer for milions of points

2 Answers2