How to extract the text $7.56 from the webpage using Selenium through Python

Question

Go to: https://www.goodrx.com/amoxicillin
Right click on $7.56 (or any price) -> copy xpath in the chrome dev tools

I've tried all these variations:

find_element(By.XPATH, '// *[ @ id = "uat-price-row-coupon-1"] / div[3] / div[1] / text()')  
find_element(By.XPATH, "//*[@id='uat-price-row-coupon-0']/div[3]/div[1]/text()")  
find_element_by_xpath("//*[@id='uat-price-row-coupon-1']/div[3]/div[1]/text()")

I also verified it works in "Try Xpath" in Firefox

But I get "no such element" from selenium with all of them.

Am I missing something?

Are you sure that the content isn't dynamically generated? You have written "go to", but I never do listen. — Alejandro, Jan 20 '20 at 21:45

Sers · Answer 1 · 2020-01-20T22:04:16.850

1

Use WebDriverWait to wait elements visibility. The website has bot protection, be ready for captcha.

import re
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC

# ...

wait = WebDriverWait(driver, 20)
with driver:
    driver.get("https://www.goodrx.com/amoxicillin")

    rows = wait.until(EC.visibility_of_all_elements_located((By.CSS_SELECTOR, 'li[data-qa="price_row"]')))
    for row in rows:
        store_name = row.find_element_by_css_selector('[class^="goldAddUnderline"]').text.strip()
        drug_price = row.find_element_by_css_selector('[data-qa="drug_price"]').text.strip()
        drug_price = re.findall(r"\d+.\d+", drug_price)[0]
        print(store_name, drug_price)

edited Jan 20 '20 at 22:04

answered Jan 20 '20 at 21:45

Sers

12,047
2
12
31

yes....i did try this...i was trying to avoid regexing the drug price and getting the text directly – Tim Boland Jan 20 '20 at 23:13
1

Xpath and text() not working directly with Selenium. You can get using JS, one way of it is in @DebanjanB answer. – Sers Jan 20 '20 at 23:23

undetected Selenium · Accepted Answer · 2020-01-20T22:11:09.573

To extract the text $7.56 as it is a text node you you have to induce WebDriverWait for the visibility_of_element_located() and you can use either of the following Locator Strategies:

Using CSS_SELECTOR:

driver.get('https://www.goodrx.com/amoxicillin')
element = WebDriverWait(driver, 20).until(EC.visibility_of_element_located((By.CSS_SELECTOR, "ul[aria-label='List of best coupons by price and pharmacy.']>li div[data-qa='drug_price']")))
print(driver.execute_script('return arguments[0].childNodes[1].textContent;', element).strip())

Using XPATH:

driver.get('https://www.goodrx.com/amoxicillin')
element = WebDriverWait(driver, 20).until(EC.visibility_of_element_located((By.XPATH, "//ul[@aria-label='List of best coupons by price and pharmacy.']/li//div[@data-qa='drug_price']")))
print(driver.execute_script('return arguments[0].childNodes[1].textContent;', element).strip())

Console Output:
```
$7.56
```

Note : You have to add the following imports :

from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC

YES! the execute_script('return arguments[0].childNodes[1] was the magic I was looking for. Thank You @DebanjanB — Tim Boland, Jan 20 '20 at 23:53

How to extract the text $7.56 from the webpage using Selenium through Python

2 Answers2