2

I am running this code

from multiprocessing.Pool import ThreadPool
from selenium import webdriver
driver = webdriver.Firefox()
driver.get(url)
elements = driver.find_elements_by_class_name("class-name")
pool = ThreadPool(4)
async = [pool.apply_async(fn_which_calls_get_attribute,(element,)) for element in elements]
results = [result.get() for result in async]

which works fine for some of the results, but throws an error of ResponseNotReady for other results. It runs as expected if I use "pool.apply" instead of the async version.

Is it a problem that I am making multiple calls to the selenium driver at once, and the error is because it cannot handle it? Or is something wrong with my parallelization?

carokann
  • 51
  • 3
  • I am not sure, but I have noticed that selenium fails when I am running the same script that uses selenium several times using a bash-script with different inputs. This does seem to imply that there is something wrong with the selenium webdriver and true parallelism – Metareven Aug 17 '15 at 13:18
  • An observastion - if you are just querying for attributes on the DOM, there are far more light weight solutions than Selenium. – Arran Aug 17 '15 at 15:47
  • This is kind of related: http://stackoverflow.com/questions/29221463/how-can-selenium-batch-many-iselementdisplayed-calls. – alecxe Aug 18 '15 at 03:20

1 Answers1

2

Just Hint that Selenium run in a single thread and in a single core system. So its not possible to exercise multi-threading over selenium webdriver . Yes you can create a separate instance and attach to another core of a multi core system.

I may not answer your question but if you are trying to do something similar good not to do.

Asit Tripathy
  • 133
  • 1
  • 9