Python: Xpath not able to locate element

Question

I am trying to get some data from a website but getting below error. It worked last night but when I rerun in today it is suddenly not able to locate the elements. Today, I tried almost I could but Couldn't resolve it.

Tools and Language - Python, Selenium, Chrome, Chromedriver, AWS Cloud 9, EC2

from selenium import webdriver
import time
from selenium.webdriver.chrome.options import Options
options = Options()
options.headless = True
driver = webdriver.Chrome(options=options)


driver.get('https://www.espncricinfo.com/series/19496/scorecard/1198235/england-vs-australia-1st-t20i-england-v-australia-2020')
time.sleep(20)
element_text = driver.find_element_by_xpath('//*[@id="main-container"]/div/div[2]/div[2]/div/div[1]/div[1]/div[1]/div[1]/div[1]/div[2]').text
print(element_text)

Error message

selenium.common.exceptions.NoSuchElementException: Message: no such element: Unable to locate element: {"method":"xpath","selector":"//*[@id="main-container"]/div/div[2]/div[2]/div/div[1]/div[1]/div[1]/div[1]/div[1]/div[2]"}

I tried below thing

1. Added and removed sleep time. Increased and decreased sleep time
2. Used full Xpath, Xpath, find by class
3. Tried to locate different elements.
4. Different pages of this.

Referred to various site still couldn't resolve. I am new to python.

Answer 1

Try this:

import time

from selenium import webdriver
from selenium.webdriver.chrome.options import Options

options = Options()
options.headless = True
driver = webdriver.Chrome(options=options)

url = 'https://www.espncricinfo.com/series/19496' \
      '/scorecard/1198235/england-vs-australia-1st-t20i-england-v-australia-2020'
driver.get(url)
time.sleep(2)
element = driver.find_element_by_xpath('//div[@class="desc text-truncate"]')
print(element.text)

Output:

1st T20I (N), Southampton, Sep 4 2020, Australia tour of England

Answer 2

To print the text 1st T20I (N), Southampton, Sep 4 2020, Australia tour of England you can use either of the following Locator Strategies:

• Using class_name and text attribute:

  print(driver.find_element_by_class_name("desc").text)
  

• Using css_selector and get_attribute():

  print(driver.find_element_by_css_selector("div.desc").get_attribute("innerHTML"))
  

• Using xpath and text attribute:

  print(driver.find_element_by_xpath("//div[@class='desc text-truncate']").text)
  

---

Ideally, to print the innerText of an element you have to induce WebDriverWait for the visibility_of_element_located() and you can use either of the following Locator Strategies:

• Using CLASS_NAME:

  print(WebDriverWait(driver, 20).until(EC.visibility_of_element_located((By.CLASS_NAME, "desc"))).text)
  

• Using CSS_SELECTOR:

  print(WebDriverWait(driver, 20).until(EC.visibility_of_element_located((By.CSS_SELECTOR, "div.desc"))).get_attribute("innerHTML"))
  

• Using XPATH:

  print(WebDriverWait(driver, 20).until(EC.visibility_of_element_located((By.XPATH, "//div[@class='desc text-truncate']"))).text)
  

• Note : You have to add the following imports :

  from selenium.webdriver.support.ui import WebDriverWait
  from selenium.webdriver.common.by import By
  from selenium.webdriver.support import expected_conditions as EC
  

You can find a relevant discussion in How to retrieve the text of a WebElement using Selenium - Python

---

Outro

Link to useful documentation:

• get_attribute() method Gets the given attribute or property of the element.
• text attribute returns The text of the element.
• Difference between text and innerHTML using Selenium