My Problem:
I am trying to scrape a webpage
#start selenium
selServ <- selenium()
# check ports
selServ$log()$stderr
# conect using port
sel <- remoteDr(browserName = "chrome", port = 4567)
# go to the URL
sel %>%
go("https://jurispub.admin.ch/publiws/pub/search.jsf")
# hit search button
sel %>%
findElement("name", "form:searchSubmitButton") %>% # find the submit button
elementClick() # click it
Next, I use a loop to get some data, at the end of the loop I let Selenium click on the next button:
i <- 1
while (i <= nr+1) {
Sys.sleep(0.5)
sel %>%
getPageSource() %>% # like read_html()
html_node("table.iceDatTbl") -> dtbl # this is the data table
sel %>%
findElement("xpath", ".//img[contains(@src, 'arrow-next')]/../../a") %>%
elementClick() # go to next page}
However, after a certain time (random amount of loops) I get this error message:
Error detected:
Response status code : 200
Selenium Status code: 10
Selenium Status summary: StaleElementReference
Selenium Status detail: An element command failed because the referenced element is no longer attached to the DOM.
Selenium message: stale element reference: element is not attached to the page document
(Session info: chrome=69.0.3497.100)
(Driver info: chromedriver=70.0.3538.16 (16ed95b41bb05e565b11fb66ac33c660b721f778),platform=Windows NT 10.0.17134 x86_64)
Please check the response with errorResponse()
Please check the content returned with errorContent()
> errorContent()
[1] "stale element reference: element is not attached to the page document\n
My approach
I think clicking the next button is causing the error, since this is the only click in the loop
sel %>%
findElement("xpath", ".//img[contains(@src, 'arrow-next')]/../../a") %>%
elementClick() # go to next page
I found this post and I think i have a similar problem: RSelenium throwing StaleElementReference error
However, I do not know how to implement it in my code.
Update
This is slightly different from from StaleElementReference Exception in PageFactory
since I've already had a findElement()
implemented before elementClick()
My question is more similar to RSelenium cannot access DOM
and I tried to implement this solution, i.e a JavaScript click:
# go to next page
sel %>% findElement("xpath", ".//img[contains(@src, 'arrow-next')]") -> resu
executeScript(sel,"arguments[0].click();", args=list(resu))
However, I still get "random" StaleElementReference errors.. (less frequently) :,( Could there anything else in my code cause this error?