I have no direct access to code but wonder if I can append my chart data points to a log file

Question

Background

A Chinese company that provides PV systems uses highcharts to present data for users. They are moving us to a newer code base Highcharts JS v7.2.0 (2019-09-03). In the old system, Highcharts JS v3.0.10 (2014-03-10), I could catch all data points in packets on my network with Wireshark while I stepped through displaying N days of data and then process the capture file to get all date, times and wattage values for my longer term analysis.

In the new system the daily data is passed in TLS as a zip file, I think this is part of the new code base feature set. It is more efficient but I can't get to see my data. The Chinese do not provide any export to CSV, Excel options (just pdf, SVG pictures), and anyway I do not really want a file to process for each day if I am working through 60 to 300 days of data. (I have emailed them previously that they might like to add export the detailed data, and add drill down from month to day view etc both of which which I now see is supported in Highchart.) I'm not holding my breath on that...

My Question

What is the easiest or best way, if any, to add some extra code in the page to append chart title, series data x&y values for each chart displayed to a file I can set at some initial point. Ideally only on first rendering to avoid getting multiple dumps of the data when the system auto-refreshes on a timer - in case I have left the recorder running!

To clarify I do not have any control of or access to the code at the server that embeds the protected TLS/zip data stream and highchair setup - it all happens in another country. To access the data values all I can think of is that I need to use some sort of console print / edit / function overload / event handler /greaseMonkey script etc I can inject/modify/add into the highchart code somehow in my browser (Safari, but I could use Firefox) after that code and data is received. I cannot see the data at all in the new system as it is behind TLS and in a zip file; all I get to see is a graph. I am not keen to add Export to CSV to the chart menu options as I do not want to have to save and work with N files...

You can see my processing and data from the old highchart based system here (where I could capture data off wifi) Best way to pull repeated table data from a pcap http file (could awk handle the disruptive breaks)?

Having dug about a little in last 14 days I am wondering if my browser can create a file stream object once and then append to that somehow when Highcharts has each data point:

 var fs = require('fs');
 var logStream = fs.createWriteStream('PVlog.txt', {flags: 'a'});
 // use {flags: 'a'} to append and {flags: 'w'} to erase and write a new file

.... somewhere in Highcharts execution log my data (date, time, power) so something like:

 logStream.write(it.heading, it.x, it.y);

Answer 1

I'm hoping I can record progress towards getting data out, the steps taken etc. I have some progress, but far from what I would want to use for capturing significant amounts of data.

For a sample of the working of the system I am referring to try clicking on any of the available sample systems currently producing power via the "Demo account" at

https://server.growatt.com/login/toViewExamlePlant

Opening js console I have found that the system creates new entries in the charts object array when a different day is displayed. Any older chart seems to get nulled to undefined (maybe this is an less than optimal use of Highcharts):

Array (3)
0 undefined
1 undefined
2 Chart {renderTo: <div class="chartDiv height1">, exporting: Object, navigation: Object, userOptions: Object, margin: [undefined, undefined, undefined, undefined], …}

So to access data in the latest current chart I have used [Highcharts.charts.length-1] to index into the charts array, which is 0 based.

For the date of data graphed my best solution so far is to examine the tooltip configuration:

 console.log(Highcharts.charts[Highcharts.charts.length-1].tooltip.options.userOptions.headerFormat)
 example output for 4 Jan 2020: 
<span style="font-size:10px">2020-01-04 {point.key}</span><table>

For data point values:

console.log(Highcharts.charts[Highcharts.charts.length-1].series[0].yData)
Seeing the end of data is awkward as long data in console truncates end in ellipsis… 
right-click > copy object       SORT OF works with paste giving:
[Log] Array (16)
0 11.9
1 10.549999
2 18.8
3 18.05
4 21
5 29.7
6 38
7 37.6
8 66.4
9 83.1
10 75.46667
11 100
12 89.666664
13 96.833336
14 94.03333
15 92.26666

For the X axis categories, showing the times of day for each reading:

console.log(Highcharts.charts[Highcharts.charts.length-1].series[0].xAxis.categories)
As before     right-click > copy object         SORT OF works with paste giving:
[Log] Array (71)
0 "09:05"
1 "09:10"
2 "09:15"
3 "09:20"
4 "09:25"
5 "09:30"
6 "09:35"
7 "09:40"
8 "09:45"
9 "09:50"
10 "09:55"
11 "10:00"
12 "10:05"
13 "10:10"
14 "10:15"
15 "10:20"

So far this is really only first steps proof that I can get at the data. It is very clunky to use and would require considerable further work!

The first step would be to create a function to call that can obtain all the data in one go, ideally formatting the data ready for combining with other days of data, tab separated. Any standard date format would be fine too, e.g. 2018-11-30 as used in the tooltip:

06/11/18    11:20   799     
06/11/18    11:25   744     
06/11/18    11:30   720     
06/11/18    11:35   681     
06/11/18    11:40   543     
06/11/18    11:45   350     
06/11/18    11:50   274     
06/11/18    11:55   230     
06/11/18    12:00   286     
06/11/18    12:05   435     
06/11/18    12:10   544     
06/11/18    12:15   899     
06/11/18    12:20   1187        
06/11/18    12:25   1575        
06/11/18    12:30   1362        
06/11/18    12:35   1423

My next aim would be to get that data appended to a single file, myPV_DataFile.txt by a single Console command, better would be automatically on "pageLoaded" or "chart_draw_completed" being signalled.

Next step forward for me:

for(q = 0; q < Highcharts.charts[Highcharts.charts.length-1].series[0].yData.length; q++) {console.log(Highcharts.charts[Highcharts.charts.length-1].series[0].xAxis.categories[q], Math.round(Highcharts.charts[Highcharts.charts.length-1].series[0].yData[q]))}

Which produces x_category,y_value pairs in the log, they do not seem to be easily copy/pasted though.

Thanks to https://github.com/edubey/browser-console-crawl/blob/master/single-story.js

... next step is to create console.save () function, and then a few lines to save the data as I want. TBH I was surprised the console would allow this much!

console.save = function (data, filename) {
    if (!data) {
        console.error('Console.save: No data')
        return;
    }

    if (!filename) filename = 'story.json'

    if (typeof data === "object") {
        data = JSON.stringify(data, undefined, 4)
    }

    var blob = new Blob([data], {
            type: 'text/json'
        }),
        e = document.createEvent('MouseEvents'),
        a = document.createElement('a')

    a.download = filename
    a.href = window.URL.createObjectURL(blob)
    a.dataset.downloadurl = ['text/json', a.download, a.href].join(':')
    e.initMouseEvent('click', true, false, window, 0, 0, 0, 0, 0, false, false, false, false, 0, null)
    a.dispatchEvent(e)
}

myPV = Highcharts.charts[Highcharts.charts.length-1].tooltip.options.userOptions.headerFormat+"\n";
for(q = 0; q < Highcharts.charts[Highcharts.charts.length-1].series[0].yData.length; q++) {myPV+=Highcharts.charts[Highcharts.charts.length-1].series[0].xAxis.categories[q] +", "+ Math.round(Highcharts.charts[Highcharts.charts.length-1].series[0].yData[q])+"\n"}
console.save(myPV, "QQsample3.txt")

Next step to strip out just the date from the headerFormat and use that as my filename, maybe. The the lot can be put in myGet function although it will still ask me to confirm each file name and save step in the UI. Sure that can be bypassed later!

Simple function can do the job for me, but is lost from console session if I refresh the page:

mySave = function () {
var myDate=Highcharts.charts[Highcharts.charts.length-1].tooltip.options.userOptions.headerFormat.substring(29,40);
myPV = myDate+"\n";
for(q = 0; q < Highcharts.charts[Highcharts.charts.length-1].series[0].yData.length; q++) {myPV+=Highcharts.charts[Highcharts.charts.length-1].series[0].xAxis.categories[q] +", "+ Math.round(Highcharts.charts[Highcharts.charts.length-1].series[0].yData[q])+"\n"}
console.save(myPV, myDate+".txt")
}

This produces a file like 2020-01-07.txt :

2020-01-07 
08:20, 16
08:25, 25
08:30, 42
08:35, 66
08:40, 67
08:45, 65
08:50, 64
08:55, 72
09:00, 112
09:05, 147
09:10, 148
09:15, 165
09:20, 223
09:25, 209
09:30, 202
09:35, 188
09:40, 176
09:45, 225
09:50, 276
09:55, 325
10:00, 300
10:05, 281
10:10, 216
10:15, 203
10:20, 205
etc.

I've not really used js much before so pleased to get to this relatively easily. Ideally I'd want the functions to persist better, and secondly to trigger the saving via some update_completed event in the Highcharts (re-)drawing or other page event.

Knowing the best, or possible, ways to approach my final 2 goals probably requires more background knowledge than I will acquire in a few days!