How can I extract information from a HTML code using Python + Selenium?

Question

I've to obtain the options that appear in this code and then put all of them into an array for later show these options in a GUI, but I don't know how to do this.

<select name="flt_technology" id="flt_technology" size="8" tabindex="1" multiple="" onchange="onChangeMessage('block','TechnicalReports');">
    <option value="3303">Aeroelastic Stability</option>
    <option value="3305">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;Aeroelastic Model</option>             
    <option value="3304">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;Aeroelastic Stability Criteria</option>
    <option value="3308">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;Aeroservoelastic Analysis</option>
    <option value="3306">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;Control Surfaces Reversal/Effectiveness</option>
    <option value="3311">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;Flutter Flight Test</option>
    <option value="3307">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;Flutter</option>
    <option value="3309">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;Flutter: Failure Conditions</option>
    <option value="3310">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;Ground Vibration Test</option>
    <option value="3710">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;Qualification Equipment Test</option>
    <option value="3588">Weight and Balance</option>
    <option value="3589">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;Center of Gravity Limits</option>
    <option value="3590">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;Leveling Means</option>
    <option value="3591">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;Loads Distribution - Weight X Cg Envelope Definition</option>
    <option value="3592">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;Weight Limits</option>
</select>

I'm new using selenium and I tried some methods but all of them had failed. Have you got an indication? — Noah, Sep 18 '19 at 11:27
hava a look at beautifulsoup 4, it's a extention for scraping html — Hans, Sep 18 '19 at 11:28
in question show code which you try to use and error message which you get. — furas, Sep 18 '19 at 11:31

score 0 · Answer 1 · answered Sep 18 '19 at 11:37

0

With <select> tag use Select class

element = self.find_element_by_id('flt_technology')
select = Select(element)
options = select.options

This will return a list of the <option> WebElements

answered Sep 18 '19 at 11:37

Guy

46,488
10
44
88

undetected Selenium · Accepted Answer · 2019-09-18T12:08:32.190

To list and print the text within the options you have to induce WebDriverWait for the visibility_of_all_elements_located() and you can use either of the following Locator Strategies:

Using CSS_SELECTOR:

select = Select(WebDriverWait(driver, 10).until(EC.element_to_be_clickable((By.CSS_SELECTOR, "select#flt_technology[name='flt_technology']"))))
print([my_elem.text for my_elem in WebDriverWait(driver, 5).until(EC.visibility_of_all_elements_located((By.CSS_SELECTOR, "select#flt_technology[name='flt_technology'] option")))])
#or
print([my_elem.text for my_elem in select.options])

Using XPATH:

select = Select(WebDriverWait(driver, 10).until(EC.element_to_be_clickable((By.XPATH, "//select[@id='flt_technology' and @name='flt_technology']"))))
print([my_elem.text for my_elem in WebDriverWait(driver, 5).until(EC.visibility_of_all_elements_located((By.XPATH, "//select[@id='flt_technology' and @name='flt_technology']//option")))])
#or
print([my_elem.text for my_elem in select.options])

Note : You have to add the following imports :

from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.support.select import Select

score -2 · Answer 3 · answered Sep 18 '19 at 11:43

Something like this:

from selenium import webdriver

driver = webdriver.Chrome()
driver.get('http://www.the_website_you_want_to_scrape.com')
select_elem = driver.find_element_by_id('flt_technology')
options = [x for x in select_elem.find_elements_by_tag_name("option")]

And if you need the values of the options:

for element in options:
    print(element.get_attribute("value"))

How can I extract information from a HTML code using Python + Selenium?

3 Answers3