selenium.common.exceptions.TimeoutException locating an element using Selenium with Python

Question

I'm trying to crawl a page that loads certain data using js request using selenium so I used "visibility_of_element_located" with a timeout. however, even setting it for 15 seconds doesn't work and I get a timeout error.

url = '''https://ukranews.com/en/news/850009-u-s-believes-russia-s-war-with-ukraine- 
will-last-until-end-of-2022-cnn'''        
options = webdriver.ChromeOptions()
options.add_argument('--headless')
options.page_load_strategy = "none"
browser = webdriver.Chrome(options=options,
                           executable_path='C:\\Users\\313-SsS\\Downloads\\Compressed\\chromedriver_win32\\chromedriver.exe',
                           )
browser.get(url)
timeout_in_seconds = 5
WebDriverWait(browser, timeout_in_seconds).until(ec.visibility_of_element_located((By.CLASS_NAME, 'article_content')))
browser.execute_script("window.stop();")
html = browser.page_source

Here is the error I got:

selenium.common.exceptions.TimeoutException: Message: 
Stacktrace:
Backtrace:
Ordinal0 [0x008E7413+2389011]

I also use the same code for another website that also loads the content using js and it works just fine.

Answer 1

There are atleast 8 elements which is identified by the locator strategy you have used:

WebDriverWait(browser, timeout_in_seconds).until(ec.visibility_of_element_located((By.CLASS_NAME, 'article_content')))

---

Solution

To wait for the visibility of the <h1> element with text as U.S. Believes Russia's War With Ukraine Will Last Until End Of 2022 - CNN you can use either of the following locator strategies:

• Using CSS_SELECTOR:

  WebDriverWait(driver, 20).until(EC.visibility_of_element_located((By.CSS_SELECTOR, "div.article_content h1")))
  

• Using XPATH:

  WebDriverWait(driver, 20).until(EC.visibility_of_element_located((By.XPATH, "//div[@class='article_content']//h1")))
  

• Note: You have to add the following imports :

  from selenium.webdriver.support.ui import WebDriverWait
  from selenium.webdriver.common.by import By
  from selenium.webdriver.support import expected_conditions as EC