0

Im very new to Python/Coding so please bare with me.

However, I'm trying to extract the text from the title of a webpage (input by the user), by scraping the 'webelement' of the page and finding its value using Selenium.

However, it keeps just returning the value 'none', instead of what I would expect to see (in this case 'BLACK BELTED WRAP COAT'.

Code can be found below:

title = driver.find_elements(By.XPATH,('/html/body/div[4]/div/div[3]/div[4]/div[1]/div[1]/form/div/div[2]/a/h2'))

//rest of code hidden but if you need more, please do let me know. (I'm new and don't want to spam)

Any idea what's causing this?

The source URL I'm entering is: https://www.riverisland.com/p/black-belted-wrap-coat-782866

This runs without error, but returns an unexpected value (as seen in below images).

enter image description here enter image description here

Appreciate it and apologies if I've missed anything. Ginge

bad_coder
  • 11,289
  • 20
  • 44
  • 72
tisGinge
  • 5
  • 1

2 Answers2

0

To print the text BLACK BELTED WRAP COAT you can use either of the following Locator Strategies:

  • Using css_selector and get_attribute("innerHTML"):

    print(driver.find_element(By.CSS_SELECTOR, "h2.product-title.ui-product-title").get_attribute("innerHTML"))

  • Using xpath and text attribute:

    print(driver.find_element(By.XPATH, "//h2[@class='product-title ui-product-title']").text)

Ideally you need to induce WebDriverWait for the visibility_of_element_located() and you can use either of the following Locator Strategies:

  • Using CSS_SELECTOR and text attribute:

    WebDriverWait(driver, 20).until(EC.element_to_be_clickable((By.CSS_SELECTOR, "button[name='accept-all']"))).click()
print(WebDriverWait(driver, 20).until(EC.visibility_of_element_located((By.CSS_SELECTOR, "h2.product-title.ui-product-title"))).text)

  • Using XPATH and get_attribute("innerHTML"):

    WebDriverWait(driver, 20).until(EC.element_to_be_clickable((By.XPATH, "//button[@name='accept-all']"))).click()
print(WebDriverWait(driver, 20).until(EC.visibility_of_element_located((By.XPATH, "//h2[@class='product-title ui-product-title']"))).get_attribute("innerHTML"))

Console Output:

BLACK BELTED WRAP COAT

  • Note : You have to add the following imports :

    from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC

You can find a relevant discussion in How to retrieve the text of a WebElement using Selenium - Python

References

Link to useful documentation:

undetected Selenium
  • 183,867
  • 41
  • 278
  • 352
  • So that has helped massively and I've gone with: driver = webdriver.Chrome() title = print(WebDriverWait(driver, 20).until(EC.visibility_of_element_located((By.XPATH, "//h2[@class='product-title ui-product-title']"))).get_attribute("innerHTML")). However, that spits out a timeout error. raise TimeoutException(message, screen, stacktrace) selenium.common.exceptions.TimeoutException: Message: Stacktrace: Backtrace: – tisGinge Nov 10 '21 at 16:33
  • @tisGinge _TimeoutException_ was due to the presence of the _Accept Cookie_ button which you need to click. Now I have added a line to handle that even. – undetected Selenium Nov 10 '21 at 17:42
0

If you are trying to find an element use find_element instead of find_elements . find_elements will return a list of webelements.

Try with below code:

Imports required for Explicit waits
from selenium.webdriver.support.wait import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.common.by import By

driver.get("https://www.riverisland.com/p/black-belted-wrap-coat-782866")

wait = WebDriverWait(driver,30)

# Click on Accept cookies
wait.until(EC.element_to_be_clickable((By.NAME,"accept-all"))).click()

title = wait.until(EC.visibility_of_element_located((By.XPATH,"//h2[@data-localize='Product_Title']")))
print(title.text)

BLACK BELTED WRAP COAT
pmadhu
  • 3,373
  • 2
  • 11
  • 23