How can i scrape the text from a located element using Selenium and Python

Question

I am trying to run the following code

from selenium import webdriver
from selenium.webdriver.chrome.options import Options
import time
options = Options()
options.add_argument("start-maximized")
options.add_argument("disable-infobars")
options.add_argument("--disable-extensions")
driver = webdriver.Chrome(options=options)
driver.get('https://theunderminejournal.com/#eu/draenor/battlepet/1155')
time.sleep(20) #bypass cloudflare
price = driver.find_element_by_xpath('//*[@id="battlepet-page"]/div[1]/table/tr[3]/td/span')
print (price)

so i can scrape the "Current Price" from the page . But this xpath location won't return the text value( i also tried the "text" varriant in the end with no success.

thanks in advance for any reply

score 2 · Accepted Answer · answered May 05 '19 at 18:28

First, use WebdriverWait to wait for the element instead of sleep.

Second, Your locator is not finding the element.

Try this,

driver.get('https://theunderminejournal.com/#eu/draenor/battlepet/1155')
price = WebDriverWait(driver,30).until(EC.visibility_of_element_located((By.XPATH,"//div[@id='battlepet-page']/div/table/tr[@class='current-price']/td/span")))

print(price.text)

To use wait import the followings,

from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.common.by import By

score 1 · Answer 2 · answered May 05 '19 at 18:24

You should wait for visibility of element before getting text. Check WebDriverWait in example below:

from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as ec
from selenium.webdriver.support.ui import WebDriverWait
rom selenium.webdriver.chrome.options import Options

options = Options()
options.add_argument("start-maximized")
options.add_argument("disable-infobars")
options.add_argument("--disable-extensions")
driver = webdriver.Chrome(options=options)

wait = WebDriverWait(driver, 20)

driver.get('https://theunderminejournal.com/#eu/draenor/battlepet/1155')
current_price = wait.until(ec.visibility_of_element_located((By.CSS_SELECTOR, ".current-price .price"))).text

print(current_price)

score 1 · Answer 3 · answered May 05 '19 at 19:50

To scrape the value of Current Price from the webpage you need to induce WebDriverWait for the visibility_of_element_located() and you can use either of the following Locator Strategies:

Using CSS_SELECTOR:

print(WebDriverWait(driver, 20).until(EC.visibility_of_element_located((By.CSS_SELECTOR, "tr.current-price td>span"))).text)

Using XPATH:

print(WebDriverWait(driver, 20).until(EC.visibility_of_element_located((By.XPATH, "//th[text()='Current Price']//following::td[1]/span"))).text)

Note : You have to add the following imports :

from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC

How can i scrape the text from a located element using Selenium and Python

3 Answers3