0

Here's the code:

from selenium import webdriver
from selenium.webdriver.common.keys import Keys
import time

drive = "******/chromedriver"
driver = webdriver.Chrome(executable_path=drive)
print(driver.execute_script("return navigator.userAgent;"))

driver.get("https://twitter.com/login")
time.sleep(3)
username = driver.find_element_by_name("session[username_or_email]")
username.send_keys('******@****.com')
time.sleep(3)
password = driver.find_element_by_name('session[password]')
time.sleep(3)
password.send_keys('*******')
password.send_keys(Keys.RETURN)
time.sleep(3)
driver.get("https://twitter.com/********/media")
time.sleep(3)

videos = driver.find_elements_by_xpath('//video[@preload="auto"]')
sources = [video.get_attribute('src') for video in videos]
print(sources)
scroll_page = driver.execute_script('return document.body.scrollHeight')

But when I repeat the "sources" and "scroll_page" I get the error: selenium.common.exceptions.StaleElementReferenceException: Message: stale element reference: element is not attached to the page document

I am only getting the first 7 url src from video tag even if the page has been scrolled to the next.

My apology if there are some prohibited in this code or the target site or the this website rules.

Many thanks in advance for the help guys!

greenman0914
  • 15
  • 4

1 Answers1

0

Add a wait function:

driver.implicitly_wait(10) #seconds

This will make the driver wait to find the element you are wanting to find, I believe the problem is when you scroll the element gets old and changes or you are scrolling before selenium gets to locate the element.

So you might want to look through inspect element with what happens when you scroll.

There are some more answers here what may help you.

Insula
  • 999
  • 4
  • 10