Rselenium and foreach with doparallel

Question

I am using Rselenium for scraping https://www.oddsportal.com. I am trying to do it in parallel since I have 50 000 URL's and want to reduce the runtime. The problem is that I am getting the same results for each clusters/nodes(3). I get 3 times each URL title. I am using code from this post Run yaml file for parallel selenium test from R or python

This is what I tried for parallel processes but it does not work the way I want it:

library(RSelenium)
library(rvest)
library(magrittr)
library(foreach)
library(doParallel)

URLsPar <- c("http://www.bbc.com/", "http://www.cnn.com", "http://www.google.com",
             "http://www.yahoo.com", "http://www.twitter.com", "http://www.oddsportal.com")

appHTML <- c()

(cl <- (detectCores() - 1) %>%  makeCluster) %>% registerDoParallel

clusterEvalQ(cl, {
  library(RSelenium)
eCap <- list(phantomjs.page.settings.userAgent 
             = "Mozilla/5.0 (Windows NT 6.1; WOW64; rv:29.0) Gecko/20120101 Firefox/29.0", phantomjs.page.settings.loadImages = FALSE, phantomjs.phantom.cookiesEnabled = FALSE, phantomjs.phantom.javascriptEnabled = TRUE)

remDr <- remoteDriver(browserName = "phantomjs", port=8910, extraCapabilities = eCap)
remDr$open()
})



myTitles <- c()
strt<-Sys.time()
ws <- foreach(x = 1:6, .packages = c("rvest", "magrittr", "RSelenium"))  %dopar%  {
  remDr$navigate(URLsPar[x])
  remDr$getTitle()[[1]]
}
end<-strt-Sys.time()

clusterEvalQ(cl, {
  remDr$close()
})

stopImplicitCluster()

Output:

[[1]]
[1] "Google"

[[2]]
[1] "Google"

[[3]]
[1] "Google"

[[4]]
[1] "Odds Portal: Odds Comparison, Sports Betting Odds"

[[5]]
[1] "Odds Portal: Odds Comparison, Sports Betting Odds"

[[6]]
[1] "Odds Portal: Odds Comparison, Sports Betting Odds"

I also tried to add remDr$navigate(url) to the cluster because I need to sign in to the page in order to proceed and get my data later in foreach function. If I add the remDr$navigate(url) to the clusterEvalQ I get only this:

[[1]]
NULL

[[2]]
NULL

[[3]]
NULL

Can someone help me how to do parallel scraping and not to get same data for each node and also to be able to login?

Code with navigate:

clusterEvalQ(cl, {
  library(RSelenium)
eCap <- list(phantomjs.page.settings.userAgent 
             = "Mozilla/5.0 (Windows NT 6.1; WOW64; rv:29.0) Gecko/20120101 Firefox/29.0", phantomjs.page.settings.loadImages = FALSE, phantomjs.phantom.cookiesEnabled = FALSE, phantomjs.phantom.javascriptEnabled = TRUE)

remDr <- remoteDriver(browserName = "phantomjs", port=8910, extraCapabilities = eCap)
remDr$open()
remDr$navigate("https://www.oddsportal.com/login/")
})

Rselenium and foreach with doparallel

0 Answers0