Selenium find_elements only works if time passes since get()

Question

Running these 2 lines together in Colab returns []:

wd.get("https://wetransfer.com/")
wd.find_elements(By.CSS_SELECTOR, 'input[type=file]')

However, running one, followed by the other returns the expected result:

[<selenium.webdriver.remote.webelement.WebElement (session="3cdfb3afbb591862e909cd406b6ac523", element="19fd31e8-710a-4b6e-8284-9a7409f12718")>,
<selenium.webdriver.remote.webelement.WebElement (session="3cdfb3afbb591862e909cd406b6ac523", element="837097d1-5735-4b24-9cb2-9d3ded3a0311")>]

Get is supposed to be blocking so not sure what is going on here.

After navigating to a url it can take some time for javascript to load all the elements on the page. — Jortega, Dec 16 '21 at 16:05

score 2 · Accepted Answer · answered Dec 16 '21 at 16:07

2

This is how basically Selenium works.
It can access web elements only after they are completely loaded.
This is why we use implicitly and explicitly waits here.
The explicitly waits are much more recommended.
So instead of

wd.get("https://wetransfer.com/")
wd.find_elements(By.CSS_SELECTOR, 'input[type=file]')

You should use something like this:

wd.get("https://wetransfer.com/")
wait = WebDriverWait(wd, 20)
wait.until(EC.presence_of_element_located((By.CSS_SELECTOR, "input[type=file]")))
wd.find_elements(By.CSS_SELECTOR, 'input[type=file]')

To use it you will have to import these imports:

from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC

answered Dec 16 '21 at 16:07

Prophet

32,350
22
54
79

This is really helpful, thanks! So what’s the behaviours of `presence_of_element_located` versus `visibility_of_all_elements_located`? – rudolfovic Dec 17 '21 at 01:10
1

`presence_of` waits for presence of element while `visibility_of` waits for visibility of element. It's recommended to use `visibility_of` since during the process of page loading the elements are being build on the page. In case Selenium catches the element on it too early state while it is already existing but still not completed it may not be clicked etc. But if element is already visibly it is clickable etc in most cases. – Prophet Dec 17 '21 at 07:59
1

`input[type=file]` element is not visibly so you can not use `visibility_of` condition for it – Prophet Dec 17 '21 at 08:00
1

`presence_of_element_located` is looking for presence of a **single** element and it returns a single element while `visibility_of_all_elements_located` is looking for visibility of elements and it return a **list** of elements matching the passed locator. You may think this method will always return you all the elements matching the passed locator, but actually it will return a list of found elements at the moment when driver founds at least one match. – Prophet Dec 17 '21 at 08:03
Thanks, that’s very thorough. So can I do anything to get the actual total number of these elements eventually present on the page? – rudolfovic Dec 17 '21 at 08:30
There is no such method provided by Selenium. What I usually do is to wait for at least 1 element existence / to be visible (according to the specific situation), then put some hardcoded sleep, like `time.sleep(1)` of a short time period it would be enough to load all the rest elements after the first one is detected and then get all the elements list by regular `driver.find_elements` method. – Prophet Dec 17 '21 at 08:41
In case you have to scroll the page in order to load more elements there is no way, just to scroll the page in order to make the elements loaded. – Prophet Dec 17 '21 at 08:42
Why in the second example after explicit wait you wrote `wd.find_elements(By.CSS_SELECTOR, 'input[type=file]')`? – Rustam-Z Feb 15 '22 at 05:50
Here I just copy-pasted the command used in the question. – Prophet Feb 15 '22 at 06:29

score 2 · Answer 2 · answered Dec 16 '21 at 22:37

On invoking get() as soon as the browser sends 'document.readyState' == "complete" to the driver, Colab executes the next line of code which doesn't find any match as the DOM Tree have completely not loaded. Hence you see Colab returning []

To locate the visible elements you need to induce WebDriverWait for the visibility_of_all_elements_located() and you can use either of the following Locator Strategies:

Using CSS_SELECTOR:

elements = WebDriverWait(driver, 20).until(EC.visibility_of_all_elements_located((By.CSS_SELECTOR, "input[type=file]")))

Using XPATH:

elements = WebDriverWait(driver, 20).until(EC.visibility_of_all_elements_located((By.XPATH, "//input[@type=file]")))

Note : You have to add the following imports :

from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC

This is really helpful, thanks! So what’s the behaviours of `presence_of_element_located` versus `visibility_of_all_elements_located`? — rudolfovic, Dec 17 '21 at 01:11

score 1 · Answer 3 · answered Dec 16 '21 at 16:02

1

You can use the following code as an example, to wait until your the site is loaded completly and ready for manipulations.

myElem = WebDriverWait(driver, delay).until(EC.element_to_be_clickable((By.CLASS_NAME , 'bs_btn_buchen')))

answered Dec 16 '21 at 16:02

Lukas Marschall

93
1
7

Selenium find_elements only works if time passes since get()

3 Answers3