How to scrape the account value and cash from investopedia with Selenium

Question

I'm trying to scrape account value and cash using the function scrapeAccCash. However, it doesn't detect the text written in their HTML code. What's the problem? This code is a minimal reproducible example.

from selenium import webdriver
from selenium.webdriver.common.by import By
from webdriver_manager.chrome import ChromeDriverManager
import time

def login():
    driver.get(r'https://www.investopedia.com/simulator/home.aspx')
    driver.implicitly_wait(10)
    driver.find_element(By.ID, 'username').send_keys('garewof922@sofrge.com')
    time.sleep(0.5)
    driver.find_element(By.ID, 'password').send_keys('Epefx8yGqFzSZL/')
    time.sleep(0.5)
    driver.find_element(By.ID, 'login').click()
    try:
        driver.find_element(By.XPATH, '//a[@class="text-h6 white--text pl-8 pr-8 v-tab"]').click()
    except:
        login()
def scrapeAccCash():
    account = driver.find_element(By.XPATH, '//div[@data-cy="account-value"]').text
    cash =  driver.find_element(By.XPATH, '//div[@data-cy="cash"]').text
    print(account, ", ", cash)
    return float(account), float(cash)

driver = webdriver.Chrome(executable_path=ChromeDriverManager().install())
login()
acc, cash = scrapeAccCash()

Answer 1

To scrape the texts as account value and cash you need to induce WebDriverWait for the visibility_of_element_located() for the elements and you can use the following locator strategies:

• Account Value:

  print(WebDriverWait(driver, 20).until(EC.visibility_of_element_located((By.XPATH, "//div[@data-cy='account-value-text']"))).text)
  

• Cash:

  print(WebDriverWait(driver, 20).until(EC.visibility_of_element_located((By.XPATH, "//div[@data-cy='cash-text']"))).text)
  

• Note : You have to add the following imports :

  from selenium.webdriver.support.ui import WebDriverWait
  from selenium.webdriver.common.by import By
  from selenium.webdriver.support import expected_conditions as EC
  

Outro: As you would be using WebDriverWait you need to disable implicitly_wait completely as mixing explicit waits and implicit waits will cause unintended consequences, namely waits sleeping for the maximum time even if the element is available or condition is true.

Warning: Do not mix implicit and explicit waits. Doing so can cause unpredictable wait times. For example, setting an implicit wait of 10 seconds and an explicit wait of 15 seconds could cause a timeout to occur after 20 seconds.