Python, Selenium : 'Element is no longer attached to the DOM'

Question

I am scraping a website, www.lipperleaders.com. I want to extract the funds detail of Singapore. I have successfully implemented drop-down selection and extracted the content of the first page appearing after submission of the options. But when I try to go to next pages (by making the code to click next button) I am getting error 'Element is no longer attached to the DOM'.

My code is about 100 lines but I can give a general idea of the flow of execution of my code:

...                    # creating driver object and all the imports
def main():
    ...
    result = find_elements_by_tag_name('span')  
    ...
    driver.find_element_by_id("ctl00_ContentPlaceHolder1_ucDataPager_btnNext").click()
    main()
main()

This code works fine for the 1st page but when main() is called again after clicking of the next button. Before this recursive method, I also tried putting this inside a loop, then also same error.

And if I write the same code like:

# some code
result = find_elements_by_tag_name('span')  
driver.find_element_by_id("ctl00_ContentPlaceHolder1_ucDataPager_btnNext").click()
# some code
driver.find_element_by_id("ctl00_ContentPlaceHolder1_ucDataPager_btnNext").click()
.
.

This code works fine w/o any error the next page loads and executes the code written after that. But I cannot write the same driver.find_element_by_id().click() for 500 pages, even I will have to repeat the rest of the code associated with each page. That's why I am trying for loop or recursion, but its not working for me.

Please let me know what is the problem with my approach.

score 8 · Accepted Answer · answered May 26 '14 at 11:45

8

The problem is that the element is being detached by some javascript. So you should make the driver wait for the element: This is done by setting implicitly_wait, see:

from selenium import webdriver

ff = webdriver.Firefox()
ff.implicitly_wait(10) # seconds
...
myDynamicElement = ff.find_element_by_id("myDynamicElement")

from http://docs.seleniumhq.org/docs/04_webdriver_advanced.jsp#implicit-waits

answered May 26 '14 at 11:45

ProfHase85

11,763
7
48
66

1

that worked but now I am getting new error `'Element not found in the cache - perhaps the page has changed since it was looked up'` – Vipul May 26 '14 at 12:55
You would need to search for the element in each iteration step of your loop. so in each iteration you do something like `next_button = driver.find_element_by_id('btnNextId') next_btn.click()` – ProfHase85 May 26 '14 at 13:23
1

yes, this is exactly what i am doing after each iteration clicking the next button to move to next page, but then also it is showing `Element not found in the cache..`. I cleared the cache of firefox then it crawled 5-6 pages and then again gave this error. – Vipul May 27 '14 at 06:44
1

Hmm this is really strange... Then you might want to start your driver with no cache as of here: https://code.google.com/p/selenium/issues/detail?id=40 – ProfHase85 May 27 '14 at 08:00
@Vipul did you find any solution? I am getting the same problem "Element not found in the cache..." – Inês Martins Jun 20 '16 at 13:39
1

@InêsMartins The link given by ProfHase85 was pretty helpful see here: https://github.com/seleniumhq/selenium-google-code-issue-archive/issues/40 start the driver with no cache – Vipul Jun 20 '16 at 13:52

score 1 · Answer 2 · answered May 26 '14 at 12:25

It seems this is an stale element exception generally it occurs when you try to find some element. Which gets loaded every time but you found it earlier, so this is a stale.

I'll suggest to use some customized method to avoid this, One of the simplest solution:

void clickOnStaleElement(String id, WebDriver driver) {
    try {
        driver.find_element_by_id(id).click();
    } catch (StaleElementReferenceException e) {
        // Trying to find element stale element
        clickOnStaleElement(id, driver);
    } catch (NoSuchElementException ele) {
        clickOnStaleElement(id, driver);
    }
 }

Python, Selenium : 'Element is no longer attached to the DOM'

2 Answers2

Linked