Screen scraping actual page not source html with R

Question

I am trying to screen scrape tennis results data (point by point data, not just final result) from this page using R.

http://www.scoreboard.com/au/match/wang-j-karlovic-i-2014/M1mWYtEF/#point-by-point;1

Using the regular R screen scraping functions like readlines(),htmlParseTree() etc I am able to scrape the source html for the page, but that does not contain the results data.

Is it possible to scrape all the text from the page, as if I were on the page in my browser and selected all and then copied?

..results data? The result of the match? – Rich Scriven Jul 19 '14 at 01:58 — Rich Scriven, Jul 19 '14 at 01:58

waternova · Accepted Answer · 2014-07-19T05:11:19.777

7

That data is loaded using AJAX from http://d.scoreboard.com/au/x/feed/d_mh_M1mWYtEF_en-au_1, so R will not be able to just load it for you. However, because both use the code M1mWYtEF, you can go directly to the page that has the data you want. Using Chrome's devtools, I was able to see that the page sends a header of X-Fsign: SW9D1eZo that will let you access that page (you get a 401 Unauthorized error otherwise).

Here is R code for getting the html that holds the data you want from your example page:

library(httr)
page_code <- "M1mWYtEF"
linked_page <- paste0("http://d.scoreboard.com/au/x/feed/d_mh_", 
                      page_code, "_en-au_1")
GET(linked_page, add_headers("X-Fsign" = "SW9D1eZo"))

edited Jul 19 '14 at 05:11

answered Jul 19 '14 at 05:01

waternova

1,472
3
16
22

+1, I learnt how the queries are handled thanks!. The following also helped to browse the chrome console (http://stackoverflow.com/questions/3200329/view-ajax-response-content-in-chrome-developer-tools) – Silence Dogood Jul 19 '14 at 06:03
waternova, you are a legend! – Liam Flynn Jul 19 '14 at 13:07

Screen scraping actual page not source html with R

1 Answers1