This is going to be an easy one for somebody, and turned into a long post for what I think is a simple question.
I am scraping a supplier's site and writing to a csv so that I don't have to manually slog through pages and pages of products. In order to get a product list I run a search with no variables and get the lot. However, if I try to limit my search to in-stock products using a dropdown option my code craps out.
Since this is a 3rd party site I cannot alter the html. Here is a slice of the html from the site (reformatted since it copied very ugly):
<tbody id="adv_search_box">
<tr>
<td class="data-name">Search in category:</td>
<td class="data-input" colspan="2">
<select name="posted_data[categoryid]" class="adv-search-select">
<option value=""> </option>
<option value="257">Hot New Arrivals</option>
<option value="252">In-Stock</option>
...
<button class="button main-button" type="submit" title="Search">
<span class="button-right"><span class="button-left">Search</span></span>
</button>
</td>
</tr>
</tbody>
When I use the following to click the search button, I get everything the supplier has with no regard to status. So this works just fine:
searchButton = driver.find_element_by_xpath('//*[@id="adv_search_box"]/tr[8]/td[2]/button/span/span')
actions = ActionChains(driver)
actions.move_to_element(searchButton)
actions.click(searchButton)
actions.perform()
Then I add code to select the "In-Stock"option in front of the button-click code:
#Select in-stock on search page
InStockSelection = driver.find_element_by_xpath('//*[@id="adv_search_box"]/tr[1]/td[2]/select/option[3]')
actions = ActionChains(driver)
actions.move_to_element(InStockSelection)
actions.click(InStockSelection)
actions.perform()
#Click button
searchButton = driver.find_element_by_xpath('//*[@id="adv_search_box"]/tr[8]/td[2]/button/span/span')
actions = ActionChains(driver)
actions.move_to_element(searchButton)
actions.click(searchButton)
actions.perform()
I get the following error:
selenium.common.exceptions.NoSuchElementException: Message: no such element: Unable to locate element: {"method":"xpath","selector":"//*[@id="adv_search_box"]/tr[8]/td[2]/button/span/span"}
One final note. I realize that this can be coded in a much, much more compact and elegant way. It is coded like this because I'm not going to look at it again until it breaks (i.e. the supplier changes their site), and if it's coded like my six-year old did it I'll be able to figure out what I did years from now.
Thank you all for your assistance.
Edit 1: It appears to me that straightforward xpath solution will not work. In pseudo-coding-babble, I think the program is focused on the drop down table and can't see the rest of the html. If I could refocus back on the page as a whole it ought to work.
Solution: Programming is to me like magic. It is often what you're not looking at that is the bit you need to watch out for. In this case, there was nothing wrong with the code to select the button. The problem was that the code to select the dropdown option wasn't working correctly, and the click at the end was changing the page focus.
The solution was to change the way the option was located and avoid the action chain.
driver.find_element_by_xpath("//select[@name='posted_data[categoryid]']/option[text()='In-Stock']").click()
Assistance was found in this article: Selenium - Python - drop-down menu option value.
Thank you, MikeJRamsey56, for convincing me to ditch the action chain, and thank you dejavu_cmd_delt for taking the time to answer.
