3

I use RSelenium to run a scraping loop which sometimes (infrequently) meets an error and then stops.

The problem for me is that when this happens and I don't check in on the RSelenium session for a while (for like half an hour or so..?), the RSelenium session closes automatically, which removes logs from the session that I want to check.

How can I stop this from happening -- or more precisely, how can I prevent the RSelenium session (and the Firefox browser opened from RSelenium) from closing when left idle for an extended time period?

The following is how I start the scraping -- I open the Firefox browser like this, then go to the URL that I want and then start scraping.

library(RSelenium)
# Running with the browser open ------------------------------------------------
rD <- RSelenium::rsDriver(port = 4454L, browser = "firefox")

remDr <- rD$client
remDr$open()

P.S. Just to clarify, it's okay that the scraping stops once in a while -- that's how I can check for loopholes that I am missing. What I need is a way for me to stop the RSelenium session from closing when left idle. Thank you in advance for any help you can give!

Hong
  • 574
  • 3
  • 10

1 Answers1

0

Found a similar issue, https://github.com/ropensci/RSelenium/issues/241

chrome_prefs = 
  list(  
   # chrome prefs
    "profile.default_content_settings.popups" = 0L,
    "download.prompt_for_download" = FALSE
    )

chrome_args = 
  c(
  # chrome command arguments
    '--headless',
    '--window-size=1200,1800',
    '-sessionTimeout 57868143'
    )

eCaps_notimeout = 
  list(chromeOptions = 
         list(
           prefs = chrome_prefs,
           args = chrome_args 
  ))


  remDr <- remoteDriver(
    browserName = "chrome",
    extraCapabilities = eCaps_withhead
  )

Further reference Is there a way too prevent selenium automatically terminating idle sessions?

Nad Pat
  • 3,129
  • 3
  • 10
  • 20