2

I'm trying to grab a table of data using read_html from the r package rvest.

I've tried the below code:

library(rvest)
  raw <- read_html("https://demanda.ree.es/movil/peninsula/demanda/tablas/2016-01-02/2")

I don't believe the above pulled the data from the table, since I see 'raw' is a list of 2:

'node:<externalptr>' and  'doc:<externalptr>'

I've tried grabbing the xpath too:

html_nodes(raw,xpath = '//*[(@id = "tabla_generacion")]//*[contains(concat( " ", @class, " " ), concat( " ", "ng-scope", " " ))]')

Any advice on what to try next?

Thanks.

cam333
  • 85
  • 1
  • 8

1 Answers1

5

This website is using angular to make a call to get the data. You can just use that call to get the raw JSON. The response is not pure JSON, so you can't just run fromJSON(url), you have to download the data and get rid of the non-JSON stuff before you parse it.

library(jsonlite)
library(httr)
url <- "https://demanda.ree.es/WSvisionaMovilesPeninsulaRest/resources/demandaGeneracionPeninsula?callback=angular.callbacks._2&curva=DEMANDA&fecha=2016-01-02"
a <- GET(url)
a <- content(a, as="text")
# get rid of the non-JSON stuff...
a <- gsub("^angular.callbacks._2\\(", "", a)
a <- gsub("\\);$", "", a)
df <- fromJSON(a, simplifyDataFrame = TRUE)

I found this by pushing F12 in Chrome and looking at the "Sources" tab. The data to fill the table had to come from somewhere... so it's just a matter of figuring out where. I was unable to use rvest to scrape the table. I'm not sure if that call to get the data was executed in R as it was in chrome... so there may have been no data for rvest to scrape.

enter image description here

cory
  • 6,529
  • 3
  • 21
  • 41
  • Awesome, thanks! Any chance you could explain where you found that call? Can it be found by using "inspect" in Chrome? – cam333 Mar 31 '16 at 14:37
  • very cool, thanks for the explanation. This will come in handy for me. – cam333 Mar 31 '16 at 14:53
  • @cory looks like an interesting solution...do you think you could help with this (http://stackoverflow.com/questions/42208566/webscrape-tables-on-websites-that-use-angularjs-using-r) ? – h.l.m Feb 13 '17 at 16:19