Cannot retrieve text from website with selenium for python

Question

i am trying to make a little program that gets the price of electricity each hour. it will retrieve the text displaying the price of the electricity.

Here is my code

from selenium import webdriver
from selenium.webdriver.common.by import By

option = webdriver.ChromeOptions()
option.add_argument('-incognito')
option.add_argument("--headless")
option.add_argument('--enable-javascript')
option.add_argument("disable-gpu")

browser = webdriver.Chrome(executable_path='chromedriver.exe', options=option)
browser.get('https://www.elektrikell.ee/')

hind = browser.find_element(by=By.XPATH, value='//*[@id="root"]/div/div/div[1]/div[2]/div[2]/b')

Once i run it, it loads up the Webdriver and then crashes with the following error

Message: no such element: Unable to locate element: {"method":"xpath","selector":"//*[@id="root"]/div/div/div[1]/div[2]/div[2]/b"}
  (Session info: headless chrome=103.0.5060.134)

it happens to everything except the HTML , when i copy that Xpath, it works fine. I enabled Javascript because i thought it might fix it, yet here i am.

Does python not support JavaScript based websites?

You may need to _wait_ for the element to load before you can find it (Yes, selenium supports running JS, regardless of Python doing so) — OneCricketeer, Aug 04 '22 at 16:20

score 0 · Accepted Answer · answered Aug 05 '22 at 23:57

To extract the price text ideally you need to induce WebDriverWait for the visibility_of_element_located() and you can use either of the following locator strategies:

Using CSS_SELECTOR:

driver.execute("get", {'url': 'https://www.elektrikell.ee/'})
print(WebDriverWait(driver, 20).until(EC.visibility_of_element_located((By.CSS_SELECTOR, "div.priceHeader div.priceElement"))).text)

Using XPATH:

driver.execute("get", {'url': 'https://www.elektrikell.ee/'})
print(WebDriverWait(driver, 20).until(EC.visibility_of_element_located((By.XPATH, "//div[@class='priceHeader']//div[@class='priceElement']"))).text)

Console output:
```
37.81
senti / kilovatt-tund
```

Note : You have to add the following imports :

from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC

You can find a relevant discussion in How to retrieve the text of a WebElement using Selenium - Python

Cannot retrieve text from website with selenium for python

1 Answers1