Big data amounts with Highcharts / Highstock (async loading)

Question

Since my data amount becomes bigger everyday (right now > 200k MySQL rows in one week), the chart is very slow at loading. I guess the async loading method is the right way to go here (http://www.highcharts.com/stock/demo/lazy-loading). I tried to implement it, but it doesn't work. So far I can provide my data with Python via URL parameters e.g. http://www.url.de/data?start=1482848100&end=1483107000, but there are several things I don't understand in the example code:

• If a period of All Data is chosen in the Navigator, then all data is provided by my server and loaded by the chart. So its the same as I what I do right now without lazy loading. Whats the difference then?
• Why there is a second getJSON() method without any URL parameter in the above mentioned example code? Its the following URL, which is empty. What do I need it for? I don't understand it: https://www.highcharts.com/samples/data/from-sql.php?callback=?

And which method to load the data is better?: This one: chart.series[0].setData(data); or the code below which I use so far:

var ohlc = [],
volume = [],
dataLength = data.length,
i = 0;
for (i; i < dataLength; i += 1) {
ohlc.push([
data[i]['0'], // date
data[i]['1_x'], // open
data[i]['2_x'], // high
data[i]['3'], // low
data[i]['4'] // close  ]);

Answer 1

The idea behind the lazy loading demo is that you fetch only the amount of points which is necessary, so if you have the data which includes 1.7 mln points, you never load so many points to the chart.

• Based on Highcharts demo. Instead of loading too many points, you request for already grouped points, you have 1.7 milion daily points, you set the navigator to 'all' (time range 1998 - 2011), you don't need daily data, so the response will include monthly points. Gains are: fetching smaller amount of data (12 * 14 = 168 instead of 1.7 mln), avoiding heavy processing data on the client side (processing, grouping, etc.) -> lower memory and cpu usage for the client, faster chart loading.

• The request for the data is in JSONP format. More information about its advantages here. So actually, url has 3 params - mandatory callback=? and optional start=?&stop=? - which indicates the points time range and its density. The first request does not have start/stop params because the server has some default values already set. After the navigator is moved, more detailed points are requested and loaded to the chart. This is the downside of the lazy loading - after the navigator is moved, your request a new set of data -> frequent data request and interruption due to the network failure.

The answer for your last question depends on if you have your data in a proper format or you don't. If you do, you can avoid looping the data on the client side and load it to the chart directly. If the format is not correct, then you have to preprocess the data, so the chart will be able to visualize them correctly. Ideally, you want the data to be in the right format after you request them - so if you can, you should do it on the server side.