Web scraping with Selenium

Question

I'm trying to scrape this website for the list of company names, code, industry, sector, mkt cap, etc in the table with selenium. I'm new to it and have written the below code:

path_to_chromedriver = r'C:\Documents\chromedriver'
browser = webdriver.Chrome(executable_path=path_to_chromedriver)

url = r'http://sgx.com/wps/portal/sgxweb/home/company_disclosure/stockfacts'
browser.get(url)

time.sleep(15)
output = browser.page_source
print(output)

However, I'm able to get the below tags, but not the data in it..

            <div class="table-wrapper results-display">
                <table>
                    <thead>
                        <tr></tr>
                    </thead>
                    <tbody></tbody>
                </table>
            </div>
            <div class="pager results-display"></div>

I have previously also tried BS4 to scrape it, but failed at it. Any help is much appreciated.

score 2 · Accepted Answer · answered Mar 26 '16 at 15:58

2

The results are in an iframe - switch to it and then get the .page_source:

iframe = driver.find_element_by_css_selector("#mainContent iframe")
driver.switch_to.frame(iframe)

I would also add a wait for the table to be loaded:

from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC

wait = WebDriverWait(driver, 10)

# locate and switch to the iframe
iframe = driver.find_element_by_css_selector("#mainContent iframe")
driver.switch_to.frame(iframe)

# wait for the table to load
wait.until(EC.visibility_of_element_located((By.CSS_SELECTOR, '.companyName')))

print(driver.page_source)

answered Mar 26 '16 at 15:58

alecxe

462,703
120
1,088
1,195

1

Had an error in switching to the iframe - now fixed, please check. – alecxe Mar 26 '16 at 16:01
1

This gives me the results I'm looking for! If you don't mind, could you kindly add a bit more comments to the codes you have written? So that I can learn more from it? Especially about the `iframe` portion and switching to it? – jake wong Mar 26 '16 at 16:15
1

@jakewong could I please refer you to the following documentation pages: http://selenium-python.readthedocs.org/navigating.html?highlight=frame#moving-between-windows-and-frames and http://selenium-python.readthedocs.org/waits.html. Should clear things up. Thanks. – alecxe Mar 26 '16 at 16:17
Will do. Thanks again for the help! – jake wong Mar 26 '16 at 16:19

score 0 · Answer 2 · edited Apr 02 '16 at 09:22

This is totally do-able. What might be the easiest is to use a 'find_elements' call (note that it's plural) and grab all of the <tr> elements. It will return a list that you can parse using find element (singular) calls on each one in the list, but this time find each element by class.

You may be running into a timing issue. I noticed that the data you are looking for loads VERY slowly. You probably need to wait for that data. The best way to do that will be to check for its existence until it appears, then try to load it. Find elements calls (again, note that I'm using the plural again) will not throw an exception when looking for elements and finding none, it will just return an empty list. This is a decent way to check for the data to appear.

Web scraping with Selenium

2 Answers2

Linked