Why can't get the hkdrates table with selenium?

Question

I want to get the hkdrates table with selenium:

from selenium import webdriver
from selenium.webdriver.chrome.options import Options

chrome_options = Options()
browser = webdriver.Chrome(options=chrome_options,executable_path='/usr/bin/chromedriver')
browser.get("https://www.bochk.com/en/investment/rates/hkdrates.html")

Now the webpage contains hkdrates opened with selenium's chrome driver:

The xpath is //*[@id="form-div"]/form/div/table[1] shown in chrome.

browser.find_elements_by_xpath('//*[@id="form-div"]/form/div/table')
[]
browser.find_elements_by_xpath('.//table') 
[]

Both of them can get nothing,how to get the hkdrates table then?

Answer 1

your table is inside a iframe , change the url for https://www.bochk.com/whk/rates/exchangeRatesHKD/exchangeRatesHKD-input.action?lang=en

from selenium import webdriver
from selenium.webdriver.chrome.options import Options

chrome_options = Options()
browser = webdriver.Chrome(options=chrome_options,executable_path='/usr/bin/chromedriver')
browser.get("https://www.bochk.com/whk/rates/exchangeRatesHKD/exchangeRatesHKD-input.action?lang=en")
table=browser.find_elements_by_xpath('//*[@id="form-div"]/form/div/table/tbody/tr')
for x in table:
    print(x.text)

Answer 2

The table you mean it's inside iframe, you need switch first. You can use a method:

.frame_to_be_available_and_switch_to_it

The iframe has an id: iframe

And for your hkdrates table, you can use css selector: .form_table.import-data.second-right

browser.get('https://www.bochk.com/en/investment/rates/hkdrates.html')

WebDriverWait(browser, 20).until(EC.frame_to_be_available_and_switch_to_it((By.ID, 'iframe')))
hkdrates_tbl = browser.find_element_by_css_selector('.form_table.import-data.second-right')
print(hkdrates_tbl.text)
browser.quit()

Following import:

from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC

Answer 3

To get the hkdrates table using Selenium as the the desired elements are within an <iframe> so you have to:

• Induce WebDriverWait for the desired frame_to_be_available_and_switch_to_it().
• Induce WebDriverWait for the desired visibility_of_all_elements_located().

• You can use the following solution:

  • Code Block:

    from selenium import webdriver
    from selenium.webdriver.common.by import By
    from selenium.webdriver.support.ui import WebDriverWait
    from selenium.webdriver.support import expected_conditions as EC
    
    options = webdriver.ChromeOptions()
    options.add_argument("start-maximized")
    driver = webdriver.Chrome(options=options, executable_path=r'C:\Utility\BrowserDrivers\chromedriver.exe')
    driver.get("https://www.bochk.com/en/investment/rates/hkdrates.html")
    WebDriverWait(driver, 20).until(EC.frame_to_be_available_and_switch_to_it((By.XPATH,"//iframe[@id='iframe' and starts-with(@src, '/whk/rates/exchangeRatesHKD/exchangeRatesHKD-input')]")))
    WebDriverWait(driver, 20).until(EC.visibility_of_all_elements_located((By.CSS_SELECTOR, "table.form_table.import-data.second-right tbody tr")))
    print(driver.find_element_by_css_selector("table.form_table.import-data.second-right tbody").text)
    

  • Console Output:

    Currency Buy Sell
    CNY 90.290000 89.360000
    CNH 90.380000 89.400000
    USD 782.250000 785.250000
    GBP 1,008.800000 1,020.400000
    JPY 721.000000 730.200000
    AUD 536.950000 543.800000
    NZD 500.200000 506.650000
    CAD 591.950000 599.100000
    EUR 868.400000 878.400000
    CHF 790.350000 797.850000
    DKK 116.150000 117.750000
    NOK 85.000000 86.400000
    SEK 80.650000 82.150000
    SGD 575.200000 579.450000
    THB 25.550000 26.550000
    BND 575.200000 579.450000
    ZAR 51.000000 52.950000
    

Here you can find a relevant discussion on Ways to deal with #document under iframe