StaleElementReferenceException: My Selenium code won't turn pages

Question

I'm trying to scrape several pages of a site with Selenium and Python, but my code is breaking over and over. I want to be able to enter the page number in the value box given at the bottom of every page. As of now my code does enter the page number but it breaks right after the new page is loaded. I've been able to scrape just the first page, and as soon as the second page loads, the code breaks.

Here's my code:

from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.common.keys import Keys

driver = webdriver.Safari()
wait = WebDriverWait(driver, 1)
driver.get("http://www.incometaxindia.gov.in/Pages/utilities/exempted-institutions.aspx")


call_names = {"Address": "Address", "State": "State", "City": "City", "Chief Commissioner of Income Tax Cadre Controlling Authority (CCIT- CCA) / DGIT (Exemptions)":"CCIT_DGIT_Exemptions", "Chief Commissioner of Income Tax (CCIT)":"CCIT", "Commissioner of Income Tax (CIT)": "CIT","Approved under Section": "Approved_under_Section", "Date of Order (DD/MM/YYYY)": "Date_of_order", "Date of Withdrawal/Cancellation (DD/MM/YYYY)":"Date_of_withdrawal", "Date of Expiry (DD/MM/YYYY)": "Date_of_Expiry", "Remarks": "Remarks"}


while True:

    for elem in wait.until(EC.presence_of_all_elements_located((By.CLASS_NAME,"faq-sub-content exempted-result"))):

        listofIDstoScrape = []

        name = elem.find_elements_by_class_name("fc-blue fquph")
        pancard = elem.find_elements_by_class_name("pan-id")
        details = driver.find_elements_by_class_name("exempted-detail")
        for i in details:
            pan = i.text

        wait.until(EC.presence_of_element_located((By.TAG_NAME, 'li')))

        for n, p, key in zip(name, pancard, details):
            main_list = {"Name": (n.text.replace(p.text,'')), "Pancard": p.text}

            for elem_li in key.find_elements_by_tag_name("li"):
                main_list[call_names [elem_li.find_element_by_tag_name('strong').text]] = elem_li.find_element_by_tag_name('span').text

            print (main_list)

    try:
        for k in range(2,10):
                myElem = WebDriverWait(driver, 1).until(EC.presence_of_element_located((By.ID, "ctl00_SPWebPartManager1_g_d6877ff2_42a8_4804_8802_6d49230dae8a_ctl00_txtPageNumber")))
                myElem.send_keys(str(k))
                myElem.send_keys(Keys.RETURN)


        print ("Page is ready!")
        break

    except TimeoutException:
            print ("Loading took too much time!")

And here's the error:

    --------------------------------------------------------------------------

    ---------------------------------------------------------------------------
    ---------------------------------------------------------------------------
StaleElementReferenceException            Traceback (most recent call last)
<ipython-input-66-aa6debbcbeae> in <module>()
     32 
     33             for elem_li in key.find_elements_by_tag_name("li"):
---> 34                 main_list[call_names [elem_li.find_element_by_tag_name('strong').text]] = elem_li.find_element_by_tag_name('span').text
     35 
     36             print (main_list)

/anaconda/lib/python3.6/site-packages/selenium/webdriver/remote/webelement.py in find_element_by_tag_name(self, name)
    230             - name - name of html tag (eg: h1, a, span)
    231         """
--> 232         return self.find_element(by=By.TAG_NAME, value=name)
    233 
    234     def find_elements_by_tag_name(self, name):

/anaconda/lib/python3.6/site-packages/selenium/webdriver/remote/webelement.py in find_element(self, by, value)
    516 
    517         return self._execute(Command.FIND_CHILD_ELEMENT,
--> 518                              {"using": by, "value": value})['value']
    519 
    520     def find_elements(self, by=By.ID, value=None):

/anaconda/lib/python3.6/site-packages/selenium/webdriver/remote/webelement.py in _execute(self, command, params)
    499             params = {}
    500         params['id'] = self._id
--> 501         return self._parent.execute(command, params)
    502 
    503     def find_element(self, by=By.ID, value=None):

/anaconda/lib/python3.6/site-packages/selenium/webdriver/remote/webdriver.py in execute(self, driver_command, params)
    309         response = self.command_executor.execute(driver_command, params)
    310         if response:
--> 311             self.error_handler.check_response(response)
    312             response['value'] = self._unwrap_value(
    313                 response.get('value', None))

/anaconda/lib/python3.6/site-packages/selenium/webdriver/remote/errorhandler.py in check_response(self, response)
    235         elif exception_class == UnexpectedAlertPresentException and 'alert' in value:
    236             raise exception_class(message, screen, stacktrace, value['alert'].get('text'))
--> 237         raise exception_class(message, screen, stacktrace)
    238 
    239     def _value_or_default(self, obj, key, default):

StaleElementReferenceException: Message: An element command failed because the referenced element is no longer available.

That's how the output looks like:

{'Name': 'INDIA INCLUSION FOUNDATION', 'Pancard': 'AABTI3598J', 'Address': 'No.250/1, 16th and 17th Cross, \nSampige Road, Malleshwaram,\nBangalore-560003.', 'State': 'KARNATAKA', 'City': 'BANGALORE', 'CCIT_DGIT_Exemptions': 'PR.CCIT BENGALURU', 'CCIT': 'CCIT(E) NEW DELHI', 'CIT': 'CIT(E) BENGALURU', 'Approved_under_Section': '12A', 'Date_of_order': '30/03/3017', 'Date_of_withdrawal': ' -  ', 'Date_of_Expiry': ' -  ', 'Remarks': ' - '}

@casualcoder hey I have updated my question with the error message. Thanks for reminding me. — Shreya Agarwal, Mar 12 '18 at 05:26
Possible duplicate of [StaleElementReference Exception in PageFactory](https://stackoverflow.com/questions/44838538/staleelementreference-exception-in-pagefactory) — undetected Selenium, Mar 12 '18 at 06:44
@DebanjanB I read that thread, but I'm still not sure how to reattach my elements to DOM. I'm a python novice. Can you help? — Shreya Agarwal, Mar 12 '18 at 07:00
Can you update the question with the exact _Manual Steps_ you are trying to _Automate_? — undetected Selenium, Mar 12 '18 at 07:01
@DebanjanB I have updated my question. Hope I've explained my problem better now. — Shreya Agarwal, Mar 12 '18 at 07:14

Shahid · Answer 1 · 2018-03-12T05:43:45.620

0

StaleElementReferenceException is thrown when:

The element has been deleted entirely.
The element is no longer attached to the DOM.

In your case, this exception is thrown for one of the find_element_by_tag_name() in Line no 35.

Make sure the element exists. If exists, try waiting for the element for a certain time before finding the element.

edited Mar 12 '18 at 05:43

answered Mar 12 '18 at 05:35

Shahid

2,288
1
14
24

Hey @Shahid, I added "wait.until(EC.presence_of_element_located((By.CLASS_NAME, "exempted-detail"))) above the main for loop. That's the class where "Span" element is located. It still throws the same error. – Shreya Agarwal Mar 12 '18 at 05:45
I also tried with this: wait.until(EC.presence_of_element_located((By.TAG_NAME, 'li'))). Still throws the same error. – Shreya Agarwal Mar 12 '18 at 05:53
Then try using `FluentWait` with ignoring the StaleElementReferenceException exception. Hope it works. – Shahid Mar 12 '18 at 06:14
FluentWait is failing as well. I still get the "StaleElementReferenceException exception". – Shreya Agarwal Mar 12 '18 at 06:42

Frank · Accepted Answer · 2018-03-13T08:19:11.130

0

I had similar problems with StaleElementReference exception.

Problem here is that after supplying the next page number and sending Keys.RETURN Selenium finds the elements where you are waiting for but this are the elements of the old page, after loading the next page this elements are not connected to the Dom anymore but replaced by the ones of the new page but Selenium is going to interact with the elements of the former page wich are no longer attached to the Dom giving a StaleElement exception.

After you pressed Keys.RETURN you have to wait till the next page is completly loaded before you start your loop again. This has to be something else then presence_of_all_elements_located((By.CLASS_NAME,"faq-sub-content exempted-result").

A good strategy for you could be waiting for the pageselector of the page you are navigating to having the class "NumericalPagerSelected". How to wait for an Element having an attribute with a specific value is described here: Using selenium webdriver to wait the attribute of element to change value

See: StaleElementException when Clicking on a TableRow in an Angular WebPage how I solved it.

edited Mar 13 '18 at 08:19

answered Mar 12 '18 at 09:16

Frank

831
1
11
23

Hey Frank, that's really helpful. I edited my code to add wait to avoid exception. But my code is still throwing the same error. Not sure if I'm doing it right. – Shreya Agarwal Mar 12 '18 at 10:01
What happens when you debugging the code and wait after pressing Keys. Return till you see that the next page is loaded? – Frank Mar 12 '18 at 10:05
So, the next page gets loaded, but then the code breaks after that. – Shreya Agarwal Mar 12 '18 at 10:06
now it is at the third for Loop ("for n, p, key in zip(name, pancard, details): I have also updated my code above to reflect the new edits. – Shreya Agarwal Mar 12 '18 at 10:14
Can you also update the stacktrace and add the output you are printing in your code? – Frank Mar 12 '18 at 10:19
First you have to press the Keys.RETURN and then wait for the element with the next page number. So you are doing it in the wrong order. I don't understand why you are doing this waiting and press key in it's own loop? – Frank Mar 12 '18 at 10:41
Oh that's because I want to loop through the pages. – Shreya Agarwal Mar 12 '18 at 10:44
hey, when I changed the order and put Keys.Return over Wait -- I get this error: URLError: – Shreya Agarwal Mar 12 '18 at 10:48
You have to stay with the original structure of your code and only add this waiting for the next element with the next page number just after the press Keys. RETURN commando. – Frank Mar 12 '18 at 10:54
Okay, that didn't work either. But, really thanks for your help, today. I'll figure something out. – Shreya Agarwal Mar 12 '18 at 11:21
What python version have you installed, when I try to run your code I'm getting a lot of errors. – Frank Mar 12 '18 at 11:49
I have installed python version 3.6 – Shreya Agarwal Mar 12 '18 at 11:54
Maybe because I'm running with Chromedriver.? – Frank Mar 12 '18 at 12:08
yeah, mine's running on safari. But you can run it on chromedriver as well. Just change the "driver = webdriver.Safari()" for chromedriver. – Shreya Agarwal Mar 12 '18 at 12:13
I added a posible wait strategy to my answer. – Frank Mar 13 '18 at 07:52
Frank, I'm unable to access that link. Would you please check if it works? – Shreya Agarwal Mar 13 '18 at 08:03
Try again please – Frank Mar 13 '18 at 08:19

StaleElementReferenceException: My Selenium code won't turn pages

2 Answers2