1

I'm trying to automate some screenscraping from the websitehttps://tulip.garden/lend

On the site, I've managed to locate the class 'lend-table', using:

options = Options()
options.headless = True
driver = webdriver.Chrome('/Users/MYNAME/Downloads/chromedriver 2', options=options)
driver.maximize_window()
driver.implicitly_wait(3)
URL = "https://tulip.garden/lend"
driver.get(URL)

driver.find_element_by_class_name('lend-table')

However, when trying to extract data deeper in this class, I am unable to. Specifically I am trying to scrape the classes (there are multiple) lend-table__row-item__cell-usd.

But when trying to find it through the find_element_by_class_name() function or the xpath method:

driver.find_element_by_xpath('//div[@class="lend-table__row-item__cell-usd"]')'

I get the same error message that Selenium is unable to locate the element:

NoSuchElementException: Message: no such element: Unable to locate element: {"method":"css selector","selector":".lend-table__row-item__cell-usd"}
  (Session info: headless chrome=103.0.5060.114)

Can someone help me out?

undetected Selenium
  • 183,867
  • 41
  • 278
  • 352
Newquant
  • 111
  • 3

1 Answers1

-2

To extract the texts from the <div> tags with class lend-table__row-item__cell-usd column you can use list comprehension and you can use either of the following locator strategies:

  • Using xpath:

    driver.get('https://tulip.garden/lend')
WebDriverWait(driver, 20).until(EC.visibility_of_element_located((By.XPATH, "//div[@class='labelled-value__value' and contains(., 'M')]")))
print([my_elem.text for my_elem in driver.find_elements(By.XPATH, "//div[@class='lend-table__row-item__cell-usd' and starts-with(., '$')]")])

  • Using xpath and inducing WebDriverWait for visibility_of_all_elements_located():

    driver.get('https://tulip.garden/lend')
WebDriverWait(driver, 20).until(EC.visibility_of_element_located((By.XPATH, "//div[@class='labelled-value__value' and contains(., 'M')]")))
print([my_elem.text for my_elem in WebDriverWait(driver, 20).until(EC.visibility_of_all_elements_located((By.XPATH, "//div[@class='lend-table__row-item__cell-usd' and starts-with(., '$')]")))])

  • Note : You have to add the following imports :

    from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC

  • Console Output:

    ['$28.98M', '$10.32M', '$8.80M', '$2.42M', '$6.25M', '$3.96M', '$11.29M', '$4.57M', '$1.25M', '$358,303.39', '$635,743.16', '$77,054.51', '$1.51M', '$645,170.77', '$428,622.60', '$8,499.13', '$819,151.65', '$467,602.40', '$22,450.03', '$22,419.35', '$1.22M', '$30,221.76', '$143,336.50', '$19,053.16', '$11,720.04', '$10,289.29', '$676,306.81', '$620,009.82', '$20,729.59', '$20,729.59', '$1.38M', '$585,494.58', '$471,111.05', '$144,437.15', '$207,759.38', '$53,695.85', '$1.31M', '$167,383.53', '$652,487.74', '$58,928.82', '$73,761.48', '$954.00', '$3,586.34', '$0.00', '$135,255.45', '$129,322.58', '$73,049.32', '$4,881.26', '$2.51M', '$745,323.40', '$1.34M', '$149,209.45', '$2.83M', '$2.71M', '$64,720.21', '$40,502.44', '$4.05', '$0.00', '$6,976.45', '$462.58', '$8,638.01', '$214.75', '$131,998.92', '$30,732.96', '$16,956.22', '$18.57', '$2,775.12', '$10.11', '$83,986.24', '$62,234.54', '$1,978.42', '$261.47']
undetected Selenium
  • 183,867
  • 41
  • 278
  • 352
  • Thank for the response. It doesn't work unfortunately. Throws this error message: TimeoutException: Message: – Newquant Jul 06 '22 at 20:44
  • @Newquant Check out the updated answer and let me know the status. – undetected Selenium Jul 06 '22 at 20:55
  • Hey thanks for the update. That's really strange that it works for you (those look like the expected output) but even after I import all the requirements I still get a timeoutexception message: 'TimeoutException: Message: '. That's with both methods. Any idea why it isn't working? THank you again. – Newquant Jul 06 '22 at 21:02
  • Some more optimization. Please retest. – undetected Selenium Jul 06 '22 at 21:07
  • Still no luck. Same error message. What system are you using? I am on a Mac using Python 3.9 – Newquant Jul 06 '22 at 22:42