9

How would i go about running RSelenium in parallel.

The following is an example using rvest in parallel

library(RSelenium)
library(rvest)
library(magrittr)
library(foreach)
library(doParallel)

URLsPar <- c("http://www.example.com/", "http://s5.tinypic.com/n392s6_th.jpg", "http://s5.tinypic.com/jl1jex_th.jpg",
         "http://s6.tinypic.com/16abj1s_th.jpg", "http://s6.tinypic.com/2ymvpqa_th.jpg")

(detectCores() - 1) %>%  makeCluster %>% registerDoParallel

ws <- foreach(x = 1:length(URLsPar), .packages = c("rvest", "magrittr", "RSelenium"))  %dopar%  {
      URLsPar[x] %>% read_html %>% as("character")}

stopImplicitCluster()
dimitris_ps
  • 5,849
  • 3
  • 29
  • 55
  • 1
    Open a separate browser for each instance using the `open` method of the `remoteDriver` class. In terms of your workflow `seleniumPipes` might be appropriate https://github.com/johndharrison/seleniumPipes – jdharrison Aug 15 '16 at 11:19
  • I have a couple of thousand urls, lets say i have 3 cores in `registerDoParallel`, whould i need to `open` 3 instances prior to the `foreach` ? I didn't know about the `seleniumPipes`! thnx – dimitris_ps Aug 15 '16 at 14:57

1 Answers1

12

On each node in the cluster start a remoteDriver:

library(RSelenium)
library(rvest)
library(magrittr)
library(foreach)
library(doParallel)

URLsPar <- c("http://www.bbc.com/", "http://www.cnn.com", "http://www.google.com",
             "http://www.yahoo.com", "http://www.twitter.com")
appHTML <- c()
# start a Selenium Server
selServ <- startServer()

(cl <- (detectCores() - 1) %>%  makeCluster) %>% registerDoParallel
# open a remoteDriver for each node on the cluster
clusterEvalQ(cl, {
  library(RSelenium)
  remDr <- remoteDriver()
  remDr$open()
})
myTitles <- c()
ws <- foreach(x = 1:length(URLsPar), .packages = c("rvest", "magrittr", "RSelenium"))  %dopar%  {
  remDr$navigate(URLsPar[x])
  remDr$getTitle()[[1]]
}

# close browser on each node
clusterEvalQ(cl, {
  remDr$close()
})

stopImplicitCluster()
# stop Selenium Server
selServ$stop()

> ws
[[1]]
[1] "BBC - Homepage"

[[2]]
[1] "CNN - Breaking News, U.S., World, Weather, Entertainment & Video News"

[[3]]
[1] "Google"

[[4]]
[1] "Yahoo"

[[5]]
[1] "Welcome to Twitter - Login or Sign up"
jdharrison
  • 30,085
  • 4
  • 77
  • 89
  • @jdharrsion: Is it possible to open multiple tabs on a single Firefox instance, using Parallel? I am aware that environment is different in all the parallel instances, but still want to know if it is possible – Bharath Mar 17 '17 at 19:05
  • How to achieve this using docker. I have tried above code, Just inserting `remoteServerAddr = "192.168.99.100", port = 4445L, browserName = "chrome"`in parentheis of remDr, but it retunrs an error An unknown server-side error occurred while processing the command. – Mislav Sep 21 '17 at 13:26