Can't make webdriver.click() or .submit() work in loop (python)

Question

I want to scrape data from an HTML table for different combinations of drop-down values via looping over those combinations. After a combination is chosen, the changes need to be submitted. This is, however, causing an error since it refreshes the page.

This is what I've done so far:

from selenium import webdriver
from selenium.webdriver.support.ui import Select
import time

browser.get('https://daten.ktbl.de/feldarbeit/entry.html')

# Selecting the constant values of some of the drop downs:
fertilizer = Select(browser.find_element_by_name("hgId"))
fertilizer.select_by_value("2") 
fertilizer = Select(browser.find_element_by_name("gId"))
fertilizer.select_by_value("193") 
fertilizer = Select(browser.find_element_by_name("avId"))
fertilizer.select_by_value("383")  
fertilizer = Select(browser.find_element_by_name("hofID"))
fertilizer.select_by_value("2") 

# Looping over different combinations of plot size and amount of fertilizer:
size = Select(browser.find_element_by_name("flaecheID"))
for size_values in size.options:
    size.select_by_value(size_values.get_attribute("value"))
    time.sleep(1)

    amount= Select(browser.find_element_by_name("mengeID"))
    for amount_values in amount.options:
        amount.select_by_value(amount_values.get_attribute("value"))
        time.sleep(1)

        #Refreshing the page after the two variable values are chosen:
        button = browser.find_element_by_xpath("//*[@type='submit']")
        button.click()
        time.sleep(5)

This leads to the error: selenium.common.exceptions.StaleElementReferenceException: Message: The element reference of <option> is stale; either the element is no longer attached to the DOM, it is not in the current frame context, or the document has been refreshed. Obviously the issue is that I did indeed refresh the document.

I tried it with .submit():

        button = browser.find_element_by_xpath("//*[@type='submit']")
        button.submit()

This leads to the error: selenium.common.exceptions.NoSuchElementException: Message: Unable to locate element: ./ancestor-or-self::form

Time seems not to be the issue here (adding time.sleep or b.implicit.wait didn't change anything).

How can I update the page without breaking from the loop?

EDIT since it was marked as duplicate: I believe that this question does not answer my problem posed here, at least not on my level of python skills, even though the error message is similar. In the other question the problem was solved via added a wait(driver, x) and an expected condition if I'm not mistaken. Here on the other hand the issue was indeed that I did not call the relevant drop-down inside AND outside the loop, as correctly stated by Zaraki Kenpachi.

score 1 · Accepted Answer · answered Feb 27 '20 at 08:54

You need to initiate size select outside the loop to collect sizes and inside loop to grab current page state.

size = Select(browser.find_element_by_name("flaecheID"))
for size in [item.get_attribute("value") for item in size.options]:
    size_select = Select(browser.find_element_by_name("flaecheID"))
    size_select.select_by_value(str(size))

    button = browser.find_element_by_xpath("//input[@type='submit']")
    button.click()

Can't make webdriver.click() or .submit() work in loop (python)

1 Answers1