RSelenium: Load a page by clicking on load more button: stop the loop if the webelement is gone

Question

I want to scrape the following page with RSelenium: https://www.letempsarchives.ch/recherche?q=%22Willy+Spuehler%22#

Before I can scrape the page I need to load all the search results on the webpage. In order to do so I have to click on the "Voir plus des résultats..."-button until the button is not visible anymore and all the search results are loaded. I tried to write a while-loop as follows:

library(RSelenium)
rD <- rsDriver(browser = "chrome", chromever = "87.0.4280.88", port = 4568L)
url <- "https://www.letempsarchives.ch/recherche?q=%22Willy+Spuehler%22#"
remote_driver <- rD[["client"]] 
remote_driver$navigate(url)

chk <- FALSE
  while(!chk){
    loadmore <- remote_driver$findElement("xpath", "//*[@class='ui fluid button huge loadMore']")
    if(length(loadmore) > 0L){
      loadmore$clickElement()
      Sys.sleep(5)
    }else
      chk <- TRUE
  }

This way I get all the search results loaded but the loop stops with the following error:

Selenium message:stale element reference: element is not attached to the page document

The loadmore-button disapears after every search results is loaded which is exactly what I need but I also need the loop to stop without an error that later my code can continue. I got the idea from here. Any help is highly appreciated.

EDIT: I made a mistake in the code that I copied in here and changed it now. (Sorry for that!) But I still get the same mistake..

score 2 · Accepted Answer · answered Jan 04 '21 at 18:24

I was able to get my code to run without stopping with the help of this post and this post. For some reason if the element is not displayed isElementDisplayed() gives me an error instead of FALSE. But with tryCatch() and suppressMessages() it runs. It is for sure not the most elegant solution but it works.

  tryCatch({
    Sys.sleep(5)
    suppressMessages({
      loadmore <- remote_driver$findElement("xpath", "//*[@class='ui fluid button huge loadMore']")
      while(loadmore$isElementDisplayed()[[1]]){
        loadmore$clickElement()  
        Sys.sleep(10)
        loadmore <- remote_driver$findElement("xpath", "//*[@class='ui fluid button huge loadMore']")
        
      }
    })
  }, 
  error = function(e) {
    NA_character_
  }
  )

score 0 · Answer 2 · answered Jan 03 '21 at 15:42

0

Why not adding simple error handler? loadmore <- try(remote_driver$findElement("xpath", "//*[@class='ui fluid button huge loadMore']")) in case of error loadmore will not be integer any more

answered Jan 03 '21 at 15:42

Ronin scholar

104
4

Thank you for answer! Unfortunately I get the same error message... – Madleina Jan 03 '21 at 18:38
Not sure, since your example is hard to reproduce, where exactly do you get the error, can you describe a little more? the "try" function should not abort execution of this loop, it will only notify you about the error - it actually prints error message unless you change it to try('your code', silent = FALSE) – Ronin scholar Jan 04 '21 at 00:37
Sorry, I'm very new to this. Always at the end when it loaded every search results. Because than the element is gone on the webpage. The code is supposed to click on the loadmore button until all the results are loaded. I want to do a search with a lot of names and therefore I want to let the search run in a loop so I don't have to do it manually. (If you have an other idea how to do this, I'm very happy to hear :) – Madleina Jan 04 '21 at 11:21

RSelenium: Load a page by clicking on load more button: stop the loop if the webelement is gone

2 Answers2