1

I am trying to scroll down an infinite scroll page and get the links of news. The problem is when I scrolled down the page for let say 100 times, and I tried to get the links, Python launched an error that says: "StaleElementReferenceException: Message: stale element reference: element is not attached to the page document". I think its because the page is get updated and scrolled page is not available any more. here is my code for scrolling the page with Selenium Webdriver: 

import urllib2
from bs4 import BeautifulSoup
from __future__ import print_function
from selenium import webdriver    #open webdriver for specific browser
from selenium.webdriver.common.keys import Keys   # for necessary browser action
from selenium.webdriver.common.by import By    # For selecting html code
import time   

driver = webdriver.Chrome('C:\\Program Files  (x86)\\Google\\Chrome\\chromedriver.exe')
driver.get('http://seekingalpha.com/market-news/top-news')
for i in range(0,100):
   driver.execute_script("window.scrollTo(0, document.body.scrollHeight);")
   time.sleep(15)
   URL = driver.find_elements_by_class_name('market_current_title')
print URL

and the code for getting the URLs

for a in URL:
    links = a.get_attribute('href')
    print(links)

I am wondering if there is any solution to settle this problem or it is possible to get URLs for this specific page with request library, as I couldn't do that.

mk_sch
  • 1,060
  • 4
  • 16
  • 31
  • This is duplicate question pl check http://stackoverflow.com/questions/27003423/python-selenium-stale-element-fix http://stackoverflow.com/questions/36013135/python-with-selenium-element-is-not-attached-to-the-page-document – DD Sharma Nov 11 '16 at 11:50
  • 1
    thank you Sharma, we both asked about the same exceptions, but my case happens when I try to scrape an infinite scroll which is different from other cases in the link . – mk_sch Nov 13 '16 at 04:27

0 Answers0