Selenium Python loop through table to print column values I get Element is no longer valid

Question

I have a page with a HTML table with 16 rows and 5 columns. I have a method to loop through the table and print out the cell values. I get the following error:

    raise exception_class(message, screen, stacktrace)
StaleElementReferenceException: Message: Element is no longer valid

The error happens on this line:

col_name = row.find_elements(By.TAG_NAME, "td")[1] # This is the Name column

My method code is:

def get_variables_col_values(self):
        try:
            table_id = self.driver.find_element(By.ID, 'data_configuration_variables_ct_fields_body1')
            #time.sleep(10)
            rows = table_id.find_elements(By.TAG_NAME, "tr")
            print "Rows length"
            print len(rows)
            for row in rows:
                # Get the columns
                print "cols length"
                print len(row.find_elements(By.TAG_NAME, "td"))
                col_name = row.find_elements(By.TAG_NAME, "td")[1] # This is the Name column                print "col_name.text = "
                print col_name.text   
        except NoSuchElementException, e:
           return False

Am i getting the element is no longer valid because the dom has updated, changed? The table has not completed in loading? How can i solve this please? Do i need to wait for the page to be fully loaded, completed?

Should i use the following WebdriverWait code to wait for page load completed?

WebDriverWait(self.driver, 10).until(lambda d: d.execute_script('return document.readyState') == 'complete')

Where about in my code should i put this line if this is required?

I ran my code again, the 2nd time it worked. The output was:

Rows length
16
cols length
6
col_name.text = 
Name
cols length
6
col_name.text = 
Address
cols length
6
col_name.text = 
DOB
...

So I need to make my code better so it works every time i run my test case. What is the best solution?

Thanks, Riaz

Answer 1

StaleElementReferenceException: Message: Element is no longer valid can mean that the page wasn't completely loaded or a script that changes the page elements was not finished running, so the elements are still changing or not all present after you start interacting with them.

You're on the right track! Using explicate waits are good practice to avoid StaleElementReferenceException and NoSuchElementException, since your script will often execute commands much faster than a web page can load or JavaScript code can finish.

Use WebDriverWait before you use WebDriver commands.

Here's a list of different "Expected Conditions" you can use to detect that page is loaded completely or at least loaded enough: http://selenium-python.readthedocs.org/en/latest/waits.html

An example of where to place the wait in your code, with an example of waiting up to 10 seconds for all 'td' elements to finish loading (you may need to use a different type of condition, amount of time, or wait for a different element, depending on what the web page as a whole is like):

from selenium.webdriver.support import expected_conditions as EC

    def get_variables_col_values(self):
            try:
                WebDriverWait(self.driver, 10).until(EC.presence_of_all_elements_located((By.TAG_NAME,'td')))
                table_id = self.driver.find_element(By.ID, 'data_configuration_variables_ct_fields_body1')
                #time.sleep(10)
                rows = table_id.find_elements(By.TAG_NAME, "tr")
                print "Rows length"
                print len(rows)
                for row in rows:
                    # Get the columns
                    print "cols length"
                    print len(row.find_elements(By.TAG_NAME, "td"))
                    col_name = row.find_elements(By.TAG_NAME, "td")[1] # This is the Name column                print "col_name.text = "
                    print col_name.text   
            except NoSuchElementException, e:
               return False