implicit or explicit waits in selenium don't work reliably by time.sleep does?

Question

I have some selenium code to log in to jupyterlab (running locally). Without waits, it fails as it tries to find the password input textbox before it exists. So I tried to use an explicit wait as that seems like the cleanest solution, but it works erratically. Implicit waits never work, it seems to block the webserver for 10 seconds before loading the page so always fails. time.sleep always works, however it loads the page and then waits the 10 seconds before entering the password which is inefficient and less clean than the selenium wait methods which as I understand wait up to 10 seconds but stop waiting as soon as the element becomes available. What have I done wrong?

from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.common.keys import Keys
from selenium.webdriver.support import expected_conditions as ec
from selenium.webdriver.support.wait import WebDriverWait

1. Explicit wait - sometimes works

   driver = webdriver.Firefox()
   driver.get(f"http://localhost:8888")
   wait = WebDriverWait(driver, 10)
   password_input = wait.until(ec.presence_of_element_located((By.ID, "password_input")))
   password = "my_password"
   password_input.send_keys(password + Keys.RETURN)
   

   sometimes I get the error:

   selenium.common.exceptions.ElementNotInteractableException: Message: Element is not reachable by keyboard

2. Implicit wait - sometime errors

   driver = webdriver.Firefox()
   driver.get(f"http://localhost:8888")
   driver.implicitly_wait(10)
   password_input = driver.find_element_by_css_selector("password_input")
   password = "my_password"
   password_input.send_keys(password + Keys.RETURN)
   

   sometimes I get the error:

   selenium.common.exceptions.ElementNotInteractableException: Message: Element is not reachable by keyboard

3. time.sleep - always works

   driver = webdriver.Firefox()
   driver.get(f"http://localhost:8888")
   time.sleep(10)
   password_input = driver.find_element_by_id("password_input")
   password = "my_password"
   password_input.send_keys(password + Keys.RETURN)
   

   While this always works, it wastes time for no reason. And the selenium wait method really should work.

What am I doing wrong?

Answer 1

While How to resolve ElementNotInteractableException: Element is not visible in Selenium webdriver? is technically a duplicate, it solves this for Java, and it always annoys me when a duplicate is for a different language, so I'll write this answer for Python.

---

The ec.presence_of_element_located(...) method just tests for the presence of an element within the document object model. It does not ensure that element is something the user can interact with. Another element might overlap it, or the element might be hidden from view for a brief moment before you call password_input.send_keys(...).

Waiting until the element is "clickable" is usually the best solution:

driver = webdriver.Firefox()
driver.get(f"http://localhost:8888")
wait = WebDriverWait(driver, 10)

# waiting until element is "clickable" means user can interact with
# it, and thus send_keys(...) can simulate keyboard interaction.
password_input = wait.until(ec.element_to_be_clickable((By.ID, "password_input")))

password = "my_password"
password_input.send_keys(password + Keys.RETURN)