How to use Selenium to click through multiple elements while avoiding Stale Element Error

Question

I'm working on making somewhat of a site map/tree (using anytree) and in order to do so, I need Selenium to find particular elements on a page (representing categories) and then systematically click through these elements, looking for new categories on each new page until we hit no more categories, ie. all leaves and the tree is populated.

I have much of this already written. My issue arises when trying to iterate through my elements list. I currently try to populate the tree depth-first, going down to the leaves and then popping back up to the original page to continue the same thing with the next element in the list. This, however, is resulting in a Stale element reference error because my page reloads. What is a workaround to this? Can I somehow open the new links in a new window so that the old page is preserved? The only fixes I have found for that exception are to neatly catch it, but that doesn't help me.

Here is my code so far (the issue lies in the for loop):

from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from anytree import Node, RenderTree

def findnodes(driver) :
    driver.implicitly_wait(5)
    try:
        nodes = driver.find_elements_by_css_selector('h3.ng-binding')
    except:
        nodes = []
    return nodes

def populateTree(driver, par) :

    url = driver.current_url
    pages = findnodes(driver)
    if len(pages)>0 :
        for page in pages:
            print(page.text)
            Node(page.text, parent=par)
            page.click()
            populateTree(driver, page.text)
            driver.get(url)

driver = webdriver.Chrome()
#Get starting page
main ='http://www.example.com'
root = Node(main)
driver.get(main)

populateTree(driver, root)

for pre, fill, node in RenderTree(root):
    print("%s%s" % (pre, node.name))

Answer 1

I haven't worked in python but have worked on java/selenium. But,I can give you the idea to overcome staleness.

Generally we will be getting the Stale Exception if the element attributes or something is changed after initiating the webelement. For example, in some cases if user tries to click on the same element on the same page but after page refresh, gets staleelement exception.

To overcome this, we can create the fresh webelement in case if the page is changed or refreshed. Below code can give you some idea.(It's in java but the concept will be same)

Example:

 webElement element = driver.findElement(by.xpath("//*[@id='StackOverflow']"));
 element.click();
 //page is refreshed
 element.click();//This will obviously throw stale exception

To overcome this, we can store the xpath in some string and use it create a fresh webelement as we go.

String xpath = "//*[@id='StackOverflow']";
driver.findElement(by.xpath(xpath)).click();
//page has been refreshed. Now create a new element and work on it
driver.findElement(by.xpath(xpath)).click();   //This works

Hope this helps you.

Answer 2

xpath variable is not suppose to be star, it an xpath to desired elements. Stale exception appears, because we click something in the browser. That requires to find all the elements each time you click. So in each loop we find all the elements driver.find_elements_by_xpath(xpath). We get a list of elements. But then we need only one of them. Therefore we take element at specific index represented idx which will range from 0 to the number of elements.

xpath = '*'
for idx, _ in enumerate(range(len(driver.find_elements_by_xpath(xpath)))):
    element = driver.find_elements_by_xpath(xpath)[idx]
    element.click()