2

I am screen scraping with phantomJS in a python/selenium setup. Specifically, I need the first N elements that match one CSS selector.

The challenge is that there are more than double as much matching elements present within the website, so the call below takes too long (speed is a major requirement for me):

targets =  WebDriverWait(driver, 1).until(EC.presence_of_all_elements_located((By.CSS_SELECTOR, css_selector)))

Is there any way to define a cutoff for found elements, like:

EC.presence_of_all_elements_located((By.CSS_SELECTOR, css_selector[0:N])))

My previous solution without CSS selectors was to copy all html in one call, convert it to text and then get the (text) elements of interest by text parsing, using a dict of words to keep/remove. This was more than 4x faster. However, this is not suitable/scalable since the dict needs to be maintained.

How to select the first n website elements matching one CSS selector in python selenium?

sudonym
  • 3,788
  • 4
  • 36
  • 61

1 Answers1

2

Try below solution to get required number of matched nodes:

n_elements = WebDriverWait(driver, 1).until(lambda driver: driver.find_elements_by_css_selector('css_selector')[:N])
Andersson
  • 51,635
  • 17
  • 77
  • 129
  • 1
    many thanks - works like a charm. I also came across another solution using positions() to define a range within selectors - in case someone else is reading this for future references https://stackoverflow.com/a/48146404/6060083 – sudonym Jan 12 '18 at 07:08