Firefox - Collecting the data used by LAB.min.js

Question

I have a device - Stiebel Eltron heat pump - home and I can track it with a web interface.

I can only access the previous month and the current month. But I would like to keep the all the data, in a numeric way rather than having it like images.

First things I need to do is get inspiration for how to connect to the page:http://stackoverflow.com/questions/5119861/record-http-form-posts-via-a-browser — J. Chomel, Mar 24 '16 at 07:45
I dont think thats the data you would want to extract. The jquery code might be doing some manipulations on the data and creating this canvas — Rajshekar Reddy, Mar 24 '16 at 14:06
The data I want is data that will allow me extrapolating temperature measures from my device. At some point it must be downloaded to my browser, in some format, right? — J. Chomel, Mar 24 '16 at 14:34
yes that is true, But the one in the image is not the actual data. I mean the one in the inspect element. You can check the Network tab for all the requests sent and data received. You can then write a small code to hit this url and then save data to file as you want. But that depends on how the server accepts requests from unknown domain — Rajshekar Reddy, Mar 24 '16 at 14:42
I am willing to find out the trick... I'll ask their tech support. — J. Chomel, Mar 24 '16 at 15:07
@Reddy, you are right. It uses angularjs.1.0.6, and I get the raw data from of of the GETs displayed thanks to Firebug. — J. Chomel, Mar 24 '16 at 18:24
Chrome also might be useful, also you can install firebug in Mozilla. — Rajshekar Reddy, Mar 24 '16 at 18:46

score 0 · Accepted Answer · edited May 23 '17 at 12:07

In the first place I must get the raw data. I know how to do it with Firefox and Firebug addin to copy/paste the data to a file... manually. It uses angularjs.1.0.6, and I get the raw data from one of the GETs displayed thanks to Firebug:

GET https://thesite/api/data/1036493/heatEnergy/today?noCacheDummyValue=1459148624909
GET https://thesite/api/data/1036493/outTemp/today?noCacheDummyValue=1459148624905
GET https://thesite/api/data/1036493/heatEnergy/lastMonth?noCacheDummyValue=1459148708330
GET https://thesite/api/data/1036493/outTemp/lastMonth?noCacheDummyValue=1459148708321

It yould be better if I could do it - by hacking directly into the LAB.min.js data? - by calling curl with my usual login post, then collect data with the GETs above

Here is how my curl login POST call looks like:

    curl -o myraw.data --data-urlencode "userName=tutu&password=xx"  'https://thesite/api/login?noCacheDummyValue=1458825357449' \
     -H 'Host: thesite' -H 'User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:45.0) Gecko/20100101 Firefox/45.0' \
     -H 'Accept: application/json, text/plain, */*' -H 'Accept-Language: en-US,en;q=0.5' -H 'Accept-Encoding: gzip, deflate, br' \
     -H 'X-Requested-With: XMLHttpRequest' \
     -H 'Content-Type: application/json;charset=utf-8' \
     -H 'Referer: https://thesite/mobile/app/app.html' \
     -H 'Content-Length: 44' -H 'Cookie: JSESSIONID=1m5aputqimremdk95fgbtmgo9'  \
     -H 'Proxy-Authorization: Basic ??' -H 'DNT: 1'   \
     -H 'Connection: keep-alive'

But I get nothing. For now, I am happy with the Firebug Copy Response Body / Paste functionality.

Now I get my raw data looking like some JSON thing I guess:

[{"values":
[{"options":null,"webId":null,"value":"8.9","date":1458842750000,"writeValue":null,"unit":null}
,"options":null,"webId":null,"value":"9.1","date":1458842451000,"writeValue":null,"unit":null}
,{"options":null, 
...
,{"options":null,"webId":null,"value":"5.5","date":1458774051000,"writeValue":null,"unit":null}],"unit":"°C"}]

Now I got my data, I thought it would be easy. But I am getting headache trying to do something with this JSON data... I know some did try already: Does LibreOffice Calc support JSON file importing/sorting?

I just have regular pattern in my JSON, and recently (re)discovered sed. So I use my new power (and how difficult it was to find I had to add a real line break in my substitute value):

cat rawdata.txt | tr -d '\n' | sed 's/},{/\
/g' | sed 's/"options":null,"webId":null,"value"://' |\
sed 's/,"writeValue":null,"unit":null//' |sed 's/,"date":/;/' |\
sed 's/\[{"values":\[{//' |sed 's/}\],"unit":"°C"}\]/\
    /' | sed 's/}\],"unit":"kWh"}\]/\
    /' > pureData.csv

And get my data like "temperature" / "timestamp":

ju@ju-HP-Compaq-dc7900-Small-Form-Factor:~/INSTALL/myproj$ cat pureData.csv
"8.9";1458842750000
"9.1";1458842451000
"9.3";1458842150000
"9.4";1458841851000
"9.7";1458841550000
"9.9";1458841251000
"10.0";1458840950000
...

... so happy at this point! When I plot it, it looks like what I have in my Firefox window!

NB: I calculate that my time axis is based on a number of milliseconds since 01.01.1970.

If anyone interested, I wrote something to retrieve data with curl there: How to use linux curl to login and retrieve data once logged?

Firefox - Collecting the data used by LAB.min.js

1 Answers1

Linked