Getting data from html table with selenium (python): Submitting changes breaks loop

Question

I want to scrape data from an HTML table for different combinations of drop down values via looping over those combinations. After a combination is chosen, the changes need to be submitted. This is, however, causing an error since it refreshes the page.

This it what I've done so far:

from selenium import webdriver
from selenium.webdriver.support.ui import Select
import time

browser.get('https://daten.ktbl.de/feldarbeit/entry.html')

# Selecting the constant values of some of the drop downs:
fertilizer = Select(browser.find_element_by_name("hgId"))
fertilizer.select_by_value("2") 
fertilizer = Select(browser.find_element_by_name("gId"))
fertilizer.select_by_value("193") 
fertilizer = Select(browser.find_element_by_name("avId"))
fertilizer.select_by_value("383")  
fertilizer = Select(browser.find_element_by_name("hofID"))
fertilizer.select_by_value("2") 

# Looping over different combinations of plot size and amount of fertilizer:
size = Select(browser.find_element_by_name("flaecheID"))
for size_values in size.options:
    size.select_by_value(size_values.get_attribute("value"))
    time.sleep(1)

    amount= Select(browser.find_element_by_name("mengeID"))
    for amount_values in amount.options:
        amount.select_by_value(amount_values.get_attribute("value"))
        time.sleep(1)

        #Refreshing the page after the two variable values are chosen:
        button = browser.find_element_by_xpath("//*[@type='submit']")
        button.click()
        time.sleep(5)

This leads to the error:selenium.common.exceptions.StaleElementReferenceException: Message: The element reference of <option> is stale; either the element is no longer attached to the DOM, it is not in the current frame context, or the document has been refreshed. Obviously the issue is that I did indeed refresh the document.

After submitting the changes and the page has loaded the results, I want to retrieve the them with:

html_source = browser.page_source
df_list = pd.read_html(html_source, match = "Dieselbedarf")

(Shout-out to @bink1time who answered this part of my question here).

How can I update the page without breaking the loop?

I would very much appreciate some help here!

score 5 · Accepted Answer · answered Feb 27 '20 at 09:29

Stale Element Reference Exception often occurs upon page refresh because of an element UUID change in the DOM.

In order to avoid it, always try to search for an element before an interaction. In your particular case, you searched for size and amount, found them and stored them in variables. But then, upon refresh, their UUID changed, so old ones that you have stored are no longer attached to the DOM. When trying to interact with them, Selenium cannot find them in the DOM and throws this exception.

I modified your code to always re-search size and amount elements before the interaction:

# Looping over different combinations of plot size and amount of fertilizer:
size = Select(browser.find_element_by_name("flaecheID"))
for i in range(len(size.options)):
    # Search and save new select element
    size = Select(browser.find_element_by_name("flaecheID"))
    size.select_by_value(size.options[i].get_attribute("value"))
    time.sleep(1)

    amount = Select(browser.find_element_by_name("mengeID"))
    for j in range(len(amount.options)):
        # Search and save new select element
        amount = Select(browser.find_element_by_name("mengeID"))
        amount.select_by_value(amount.options[j].get_attribute("value"))
        time.sleep(1)

        #Refreshing the page after the two variable values are chosen:
        button = browser.find_element_by_xpath("//*[@type='submit']")
        button.click()
        time.sleep(5)

Try this? It worked for me. I hope it helps.

Getting data from html table with selenium (python): Submitting changes breaks loop

1 Answers1

Linked