R rvest Can't Get html_node

Question

I have some experience using the rvest package to scrape data I need from the web, but am hitting an issue with this page:

https://www.nytimes.com/interactive/2020/us/covid-college-cases-tracker.html

If you scroll down a bit, you'll see a portion where all the schools are located.

I would like to have the school, cases and location data. I should note that someone asked on the NYT GitHub publishing this as a csv and they recommended that the data is all in the page and can just be pulled from there. Thus, I think it is OK to scrape from this page.

But I can't get it to work. Let's say I just want to start with a simple selector for that first school. I use the inspector to find the xpath.

I get no results:

library(rvest)

URL <- "https://www.nytimes.com/interactive/2020/us/covid-college-cases-tracker.html"
pg <- read_html(URL)

# xpath copied from inspector
xpath_first_school <- '//*[@id="school100663"]'

node_first_school <- html_node(pg, xpath = xpath_first_school)

> node_first_school
{xml_missing}
<NA>

I get {xml_missing}.

I obliviously have a lot more to do to generalize this and gather data for all schools, but with web scraping, I usually try to start simple and specific and then broaden out. But even my simple test isn't working. Any ideas?

Answer 1

Setting up Rselenium can take some time. First you have to download chromedriver (https://chromedriver.chromium.org/), select the version your current chrome is the closest too. Then unpack it to your R working directory.

I tried using a package called decapitated where it can scrape javascript rendered websites but because this website contains "show more" which needs to be physically clicked before all data are shown, I had to use Rselenium to "click" that before I get the page source then use rvest for parsing

Code:

library(rvest)
library(tidyverse)
library(RSelenium)

url <- "https://www.nytimes.com/interactive/2020/us/covid-college-cases-tracker.html"

driver <- rsDriver(browser = c("chrome"), chromever = "85.0.4183.87", port = 560L)
remote_driver <- driver[["client"]] 
remote_driver$navigate(url)

showmore <- remote_driver$findElement(using = "xpath", value = "//*[@id=\"showall\"]/p")
showmore$clickElement()

test <- remote_driver$getPageSource()

school <- read_html(test[[1]]) %>%
  html_nodes(xpath = "//*[contains(@id, \"school\")]/div[2]/h2") %>%
  html_text() %>%
  as.tibble()

case <- read_html(test[[1]]) %>%
  html_nodes(xpath = "//*[contains(@id, \"school\")]/div[3]/p") %>%
  html_text() %>%
  as.tibble() 

location <- read_html(test[[1]]) %>%
  html_nodes(xpath = "//*[contains(@id, \"school\")]/div[4]/p") %>%
  html_text() %>%
  as.tibble() 

combined_table <- bind_cols(school,case = case[2:nrow(case),],location = location[2:nrow(location),]) 
names(combined_table) <- c("school", "case", "location")

combined_table %>% view()

Output:

# A tibble: 913 x 3
   school                                      case  location              
   <chr>                                       <chr> <chr>                 
 1 University of Alabama at Birmingham*        972   Birmingham, Ala.      
 2 University of North Carolina at Chapel Hill 835   Chapel Hill, N.C.     
 3 University of Central Florida               727   Orlando, Fla.         
 4 University of Alabama                       568   Tuscaloosa, Ala.      
 5 Auburn University                           557   Auburn, Ala.          
 6 North Carolina State University             509   Raleigh, N.C.         
 7 University of Georgia                       504   Athens, Ga.           
 8 Texas A&M University                        500   College Station, Texas
 9 University of Texas at Austin               483   Austin, Texas         
10 University of Notre Dame                    473   Notre Dame, Ind.      
# ... with 903 more rows

Hope this works for you!

Answer 2

So I am going to provide an answer here which violates a very important rule described here and is generally an ugly solution. But it is a solution that allows us to avoid having to use Selenium.

To use html_nodes on this, we need to initiate JS actions which requires Selenium. @KWN's solution seems to work on their machine, but I can't get the chromedriver to work on mine. I can get almost there using Docker with Firefox or Chrome, but can't get the result. So I would check that solution out first. And if that fails, give this a shot. Pretty much, this site has the data I need exposed as a JSON. So I pull the text of the site which I use a regex to isolate the JSON and then jsonlite to parse.

library(jsonlite)
library(rvest)
library(tidyverse)

url <- "https://www.nytimes.com/interactive/2020/us/covid-college-cases-tracker.html"

html_res <- read_html(url)

# get text
text_res <- html_res %>% 
  html_text(trim = TRUE)

# find the area of interest
# find the area of interest
data1 <- str_extract_all(text_res, "(?<=var NYTG_schools = ).*(?=;)")[[1]]

# get json into data frame
json_res <- fromJSON(data1)

# did it work?
glimpse(json_res)

Rows: 1,515
Columns: 16
$ ipeds_id    <chr> "100663", "199120", "132903", "100751"...
$ nytname     <chr> "University of Alabama at Birmingham",...
$ shortname   <chr> "U.A.B.", "North Carolina", "Central F...
$ city        <chr> "Birmingham", "Chapel Hill", "Orlando"...
$ state       <chr> "Ala.", "N.C.", "Fla.", "Ala.", "Ala."...
$ county      <chr> "Jefferson", "Orange", "Orange", "Tusc...
$ fips        <chr> "01073", "37135", "12095", "01125", "0...
$ lat         <dbl> 33.50199, 35.90491, 28.60258, 33.21402...
$ long        <dbl> -86.80644, -79.04691, -81.20223, -87.5...
$ logo        <chr> "https://static01.nyt.com/newsgraphics...
$ infected    <int> 972, 835, 727, 568, 557, 509, 504, 500...
$ death       <int> 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0,...
$ dateline    <chr> "n", "n", "n", "n", "n", "n", "n", "n"...
$ ranking     <int> 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12,...
$ medicalnote <chr> "y", NA, NA, NA, NA, NA, NA, NA, NA, N...
$ coord       <list> [<847052.5, -406444.3>, <1508445.93, ...