0

I am trying to scrape data out of a HighCharts. I've tried a diverse amount of things following similar questions/problems on StackOverflow but I can't seem to crack it here.

It's on this URL: https://www.bustabit.com/user/9mins

So some solutions suggest to use the console command Highcharts.charts[..]. That does not work here. It's possible to just grab "highcharts-series-group". However the resulting data:

M 3.7745098039216 79.5788071428571 L 7.5871459694989 94.70737857142853 L 11.399782135076 48.41395 L.....

This is transformed data (data -> transformation -> SVG coordinates?). So the question would be how to get the data pre-transformed.

A different StackOverflow question/answer managed to do the reverse transformation by finding the code in a local js file in the page sources. I've found something similar in the page sources but very difficult to decipher because of it's length.

Am I to find the answer in this file to reverse engineer the numbers or is there a way to get the data pre-transformation in a different way?

Lud
  • 36
  • 1
  • 8
  • It seems that this data is hidden deliberately (Highcharts global object is hidden). Highcharts support team is not able to help you get this data. In this case, the best option is to contact the app creator. – Sebastian Wędzel Mar 30 '20 at 12:17
  • Hi Sebastian, thanks for having a look. It's rather odd it's hidden because all information is publicly available game per game! – Lud Mar 30 '20 at 20:45

2 Answers2

2

You can do this from a browser's developer console. Note that any legality of using this data falls upon you - data licensing needs to be looked into for the site you want to extract the data from. In your browser's developer console command line type:

console.log($('#chartDiv').highcharts().series)

And then press enter. The #chartDiv is the DIV's ID that contains the chart. Typically this is the one just above a DIV with id="highcharts-information-region-0" or something similar. This will spit out all the series that are shown in the chart. In each of the series objects returned there is a property called data which is what you are after.

wergeld
  • 14,332
  • 8
  • 51
  • 81
  • This does not really help. The data is not accessible in this way. I also already mentioned it in the question that you can't simply grab the data. There is some data in the highcharts-series-group part but this is transformed and is not what I am looking for. – Lud Mar 29 '20 at 07:03
  • @Lud, I am not sure what it is you are looking for then. That command above will give you the non-transformed data - it is literally the data used to build the graph and is not transformed. I am unable to get that page to load a graph because i have no permissions. – wergeld Mar 29 '20 at 17:43
  • I appreciate you are trying to help, but if you can not test your solution? There is no data property. There is a d property as I mentioned above but it only contains SVG coordinates and not real data. – Lud Mar 29 '20 at 18:18
  • @Lud, can you try and run that code snippet? Change the `chartDiv` text to the DIV ID containing the chart you want to pull data from. My code does not spit out the SVG shape - it returns the series and the data in each series for the chart you specify. – wergeld Mar 29 '20 at 18:47
  • it does not work. So basically the div with the chartDiv has one child with class="highcharts-root" and that contains a lot of (x-axis y-axis etc.) and a with a highcharts-series-group class. This one then contains the transformed data in a with class: highcharts-series highcharts-series-0 highcharts-line-series highcharts-color-0 – Lud Mar 29 '20 at 19:08
  • @Lud, that seems correct. I am on another machine and can load this site now. I will get back to you. – wergeld Mar 29 '20 at 19:18
  • @Lud, that site is interesting. I get an error when trying to run the code that "highcharts is not a function" when I do it against `highcharts-f0i9jun-210`. – wergeld Mar 29 '20 at 19:26
  • the only alternative I see is using Selenium to hover over the chart and grab the data from the small popup? But I'm not that experienced at selenium to see the viability. – Lud Mar 30 '20 at 20:46
  • @wergeld This works for me (I can see the data attributes), but how do I export them from the console? – the.real.gruycho Aug 06 '22 at 11:53
0

In the end there was no way to fetch the pure data through some hidden method or file or script. However I did notice that sufficient information was visible in the tooltip. If you hover over the chart, it generates a tooltip per datapoint which gives you the the net cumulative profit and the bet placed etc.

What I did was use Selenium to find the y-axis label on the chart. Then use move_to_element and horizontal move_by_offset to hover over the chart and generate the tooltip. I just took some guesses and found that jumping with offset 7 generates all the tooltips in one go without doubles or missing any. The website also refreshes the chart so I catch exceptions for that to reset everything and continue again. Then after 107 datapoints, hit previous page and restart!

Lud
  • 36
  • 1
  • 8