How to extract option from "" by selenium driver?

Question

I want to use selenium and python to crawl this site : https://ntrl.ntis.gov/NTRL

However when i want to change the year of drop-down list, it can't work.

Here are its HTML:

<div id="advSearchForm:FromYear" class="ui-selectonemenu ui-widget ui-state-default ui-corner-all" style="min-width: 63px;">
    <div class="ui-helper-hidden-accessible">
        <input id="advSearchForm:FromYear_focus" name="advSearchForm:FromYear_focus" type="text" autocomplete="off" role="combobox" aria-haspopup="true" aria-expanded="false" readonly="readonly" aria-autocomplete="list" aria-owns="advSearchForm:FromYear_items" aria-activedescendant="advSearchForm:FromYear_0" aria-describedby="advSearchForm:FromYear_0" aria-disabled="false">
    </div>
    <div class="ui-helper-hidden-accessible">
        <select id="advSearchForm:FromYear_input" name="advSearchForm:FromYear_input" tabindex="-1">
            <option value="*" selected="selected">&lt;1900</option>
            <option value="1900">1900</option>
            <option value="1901">1901</option>
            <option value="1902">1902</option>
            <option value="1903">1903</option>
        </select>
    </div>
    <label id="advSearchForm:FromYear_label" class="ui-selectonemenu-label ui-inputfield ui-corner-all">&lt;1900</label>
    <div class="ui-selectonemenu-trigger ui-state-default ui-corner-right">
        <span class="ui-icon ui-icon-triangle-1-s ui-c"/>
    </div>
</div>

Here are my code:

select = Select(driver.find_element_by_xpath(".//div[@id='advSearchForm:FromYear']/div[2]/select"))
select.select_by_value("1902") 

But it get exception:

Element is not currently visible and may not be manipulated

I tried to use js script:

driver.execute_script("document.getElementById('advSearchForm:FromYear_input').options[2].selected = 'true'")

But it also don't work,I test that select.select_by_value(xxx) can be used on other drop-down list, so it may be the trouble of <div class="ui-helper-hidden-accessible">, so how can i deal with it?

Answer 1

I would suggest using the click event to click on the element (the Select element with the id of "advSearchForm:FromYear_input") first, then the ExplicitWait event to wait until the element is visible, then you should be able to change the year using the select_by_value method.

Also I would steer away from using XPath and use a CSS selector instead, even better would be to create a Page Object Model to reduce the work required to keep your tool working in the future should the page be updated.

Sorry that I cannot be much more help, I am not that familiar with python.

You could also refer to this question.

Edit

It looks like that are using the option items inside the select as the master list and the actual selection takes place inside another element further down the page. This element is dynamically built in Javascript, so my suggestion in the comments will not work.

I have hacked together a working application in C# to give you an idea of what you need to do:

private static void Main(string[] args)
{
    // ':' has a special meaning in CSS selectors so we need to escape it using \\
    const string dropdownButtonSelector = "div#advSearchForm\\:datePublPanel div.ui-selectonemenu-trigger";
    // {0} is a placeholder which is used to insert text during runtime
    const string dynamicallyBuiltListItemSelectorTemplate = "ul#advSearchForm\\:FromYear_items li[data-label=\"{0}\"]";
    // Rather than being a constant this value will be determined at runtime
    const string valueToSelect = "1902";

    // Setup driver and wait
    ChromeDriver driver = new ChromeDriver();
    WebDriverWait wait = new WebDriverWait(driver, TimeSpan.FromSeconds(5));

    // Load page
    driver.Navigate().GoToUrl("https://ntrl.ntis.gov/NTRL/");
    // Wait until the first (index 0) dropdown list button inside the publication date dive is deemed "clickable"
    wait.Until(ExpectedConditions.ElementToBeClickable(driver.FindElementsByCssSelector(dropdownButtonSelector)[0]));

    Console.WriteLine("Element is visible");

    // Open the dropdown list
    driver.FindElementsByCssSelector(dropdownButtonSelector)[0].Click();

    Console.WriteLine("Dropdown should be open");

    // Select the element from the dynamic Javascript built list
    string desiredValueListItemSelector = string.Format(dynamicallyBuiltListItemSelectorTemplate, valueToSelect);
    driver.FindElementByCssSelector(desiredValueListItemSelector).Click();

    Console.WriteLine($"Selected value {valueToSelect} using selector: {desiredValueListItemSelector}");
    Console.ReadLine();

    driver.Close();
}

==========================================================================

Edit2

Include python answer, I have never written python before but this appears to work. I would strongly suggest looking at some of the links I posted above about using the PageObject model and Explicit waits, as well as avoiding the use of XPATH selectors though.

from selenium import webdriver
from time import sleep

# Set the year to select
fromYearToSelect = "1902"

# Create the driver and load the page
driver = webdriver.Chrome("C:\chromedriver_win32\chromedriver.exe")
driver.get("https://ntrl.ntis.gov/NTRL/")

# Find and click the "From" dropdown elems[1] is the "To" dropdown
elems = driver.find_elements_by_css_selector("div#advSearchForm\\:datePublPanel div.ui-selectonemenu-trigger")
elems[0].click()

# Select the year
driver.find_element_by_css_selector("#advSearchForm\\:FromYear_items li[data-label='{0}']".format(fromYearToSelect)).click()

# Wait to see the results (we should be using an Explicit Wait  here)
sleep(2)

# Close the driver
driver.close()