0

I am trying to find all elements of a certain kind on a webpage with Selenium (python). For simplicity, let's say their id is elem_id. I am using the following code snippet to do so:

all_elements = driver.find_elements_by_id("elem_id")
print(str(len(all_elements)))

I know that there are ~3000 of this kind of element on the webpage in question, but whenever I print the length of all_elements, it always prints 1000.

It definitely finds the right kind of element (I checked), but somehow it doesn't find all of them at once. It also selects the 1000 elements at random, meaning that it neither picks the first 1000 or the last 1000 exclusively. I tried finding out whether there's a cap on how many elements Selenium can find, but there doesn't appear to be a max amount of 1000.

Does anyone know why Selenium only finds 1000 elements at a time? What am I doing wrong? Thank you very much!

undetected Selenium
  • 183,867
  • 41
  • 278
  • 352
Nooyi
  • 316
  • 2
  • 12
  • 1
    They could possibly be in an iframe – Noah Jan 03 '20 at 00:23
  • can you post the html, so we can debug? – Ramon Medeiros Jan 03 '20 at 00:25
  • 3
    some pages uses "lazy loading" technic and they load elements when you scroll page - so at start they may have only 1000 elements but after scrolling they may have more. So you may need to scroll page with Selenium too. Some pages are even worst because they can remove from page elements which you don't see and they may have always only 1000 elements when you scroll page - so you can't scroll to the bottom and get all elements but you have to scroll partially and get only new elements. – furas Jan 03 '20 at 00:50

1 Answers1

1

Fundamentally, you are seeing the correct behavior. Though you are aware there are almost ~3000 of this particular type of element within the webpage, but:

  • All those elements with id as elem_id may not be visible within the Viewport

You can find a relevant detailed discussion in How does Selenium click on elements that are 50% on screen and 50% not on screen?

  • Some of those elements may be within <iframe> / <frame> tags and WebDriver instance may not have the visibility of those elements from the Top Level View.

You can find a relevant detailed discussion in Ways to deal with #document under iframe

  • Some of those elements may remain invisible through Lazy Loading

You can find a relevant detailed discussion in How to click on Load More button within Google Trends and print all the titles through Selenium and Python

Hence you see only ~1000 of those elements out of ~3000 odd elements.

undetected Selenium
  • 183,867
  • 41
  • 278
  • 352