0

I am trying to collect a number of links from a website.

For example I have the following and my idea was to collect the link where it says leer más which is where I get the xpath from.

url = "https://www.fotocasa.es/es/alquiler/viviendas/madrid-capital/todas-las-zonas/l/181"
x <- GET(url, add_headers('user-agent' = desktop_agents[sample(1:10, 1)]))
x %>% 
  read_html() %>% 
  html_nodes(xpath = '//*[@id="App"]/div[2]/div[1]/main/div/div[3]/section/article[1]/div/a/p/span[2]')

This gives me the following but not the link:

{xml_nodeset (1)}
[1] <span class="re-CardDescription-link">Leer más</span>

Additionally, I thought about collecting all links:

x %>% 
  read_html() %>% 
  html_nodes("a") %>% 
  html_attr("href")

This gives me a lot of links but not the links to the individual webpages I want.

I would like to have a list of links such as:

https://www.fotocasa.es/es/alquiler/vivienda/madrid-capital/aire-acondicionado-calefaccion-terraza-trastero-ascensor-amueblado-internet/162262978/d

https://www.fotocasa.es/es/alquiler/vivienda/madrid-capital/aire-acondicionado-calefaccion-trastero-ascensor-amueblado/159750574/d

https://www.fotocasa.es/es/alquiler/vivienda/madrid-capital/aire-acondicionado-calefaccion-jardin-zona-comunitaria-ascensor-patio-amueblado-parking-television-internet-piscina/162259162/d
QHarr
  • 83,427
  • 12
  • 54
  • 101
user113156
  • 6,761
  • 5
  • 35
  • 81
  • 1
    You may need to use `RSelenium` to get the links as webpage use java script to load the contents. Using `rvest` you can get links for only the first few houses – Nad Pat Apr 09 '22 at 15:10
  • Thanks, your previous answer here ( https://stackoverflow.com/questions/71802822/rvest-collecting-page-numbers-from-a-website/71807216?noredirect=1#comment126896133_71807216 ) helps me to load the whole page `webElem$sendKeysToElement(list(key = "up_arrow"))` – user113156 Apr 09 '22 at 15:15

1 Answers1

1

Those links are stored inside a JavaScript object within a script tag. You can regex out the string defining that object, do some unescapes to enable jsonlite to parse, then apply a custom function to extract just the urls of interest to the json object

library(rvest)
library(jsonlite)
library(magrittr)
library(stringr)
library(purrr)

link <- 'https://www.fotocasa.es/es/alquiler/viviendas/madrid-capital/todas-las-zonas/l/181'
p <- read_html(url) %>% html_text()
s <- str_match(p, 'window\\.__INITIAL_PROPS__ = JSON\\.parse\\("(.*)".*?;')[,2]
data <- jsonlite::parse_json(gsub('\\\\\\"', '\\\"', gsub('\\\\"', '"', s)))
links <- purrr::map(data$initialSearch$result$realEstates, ~ .x$detail$`es-ES` %>% url_absolute(link))
QHarr
  • 83,427
  • 12
  • 54
  • 101