Intention:
My aim is to scrape football data from whoscored.com. Match pages (see example here) contain a timeline view (div id = match-centre-timeline) which has timeline handles in turn (div class = timeline-handle). The handles can be dragged and dropped so to set the timeline to show the match stats for. For example, I'd like to set the lower timeline bound to represent 10 minutes and the upper to represent 30 minutes.
Setup & Hypothesis:
I'm using selenium with chrome on OSX 10. The code below moves the lower timeline handle but throws a StaleElementReferenceException. My belief is that the page application becomes confused by the sudden drag and drop.
Question:
Is there any way to simulate a slower mouse drag? Is this even the problem? How can this be overcome? Thanks in advance!
Python 3 code:
url = "https://www.whoscored.com/Matches/1190514/Live"
time_to_wait = 20
wanted_element_id = "match-centre-timeline"
opts = ChromeOptions()
opts.add_experimental_option("detach", True)
opts.add_argument("disable-infobars")
opts.add_argument("disable-notifications")
driver = webdriver.Chrome(executable_path="/Users/david/Dropbox/Code/gitCode/driver/chromedriver", chrome_options=opts)
driver.set_page_load_timeout(time_to_wait)
example_loaded = bool(True)
while example_loaded:
try:
driver.get(url)
WebDriverWait(driver,time_to_wait).until(EC.presence_of_element_located((By.ID, wanted_element_id)))
print("[STATUS]\tFound " + wanted_element_id + ".")
example_loaded= False
except TimeoutException:
print("[WARNING]\tCannot find the " + wanted_element_id + ". Retrying.")
tlh = driver.find_element_by_xpath("//*[@id='match-centre-timeline']/div[1]/div[2]/div[2]/div[2]/div[1]/span")
tuh = driver.find_element_by_xpath("//*[@id='match-centre-timeline']/div[1]/div[2]/div[2]/div[2]/div[2]/span")
driver.execute_script("arguments[0].scrollIntoView();", tlh)
actions = ActionChains(driver)
# move to the lower timeline handle
actions.move_to_element(tlh)
actions.perform()
print("[NOTE]\t\tLower handle stats: x:{}, y:{} and h:{}px, w:{}px.".format( tlh.location['x'], tlh.location['y'], tlh.size['height'], tlh.size['width'] ))
print("[NOTE]\t\tUpper handle stats: x:{}, y:{} and h:{}px, w:{}px.".format( tuh.location['x'], tuh.location['y'], tuh.size['height'], tuh.size['width'] ))
# let's click on the lower timeline handle
try:
actions.click_and_hold(tlh)
actions.drag_and_drop_by_offset(tlh, 100, 0)
actions.perform()
print("[NOTE]\t\tLower handle stats: x:{}, y:{} and h:{}px, w:{}px.".format( tlh.location['x'], tlh.location['y'], tlh.size['height'], tlh.size['width'] ))
print("[NOTE]\t\tUpper handle stats: x:{}, y:{} and h:{}px, w:{}px.".format( tuh.location['x'], tuh.location['y'], tuh.size['height'], tuh.size['width'] ))
except StaleElementReferenceException:
print("[WARN]\tThrew and caught a StaleElementReferenceException.")
Console Error:
[STATUS] Found match-centre-timeline.
[NOTE] Lower handle stats: x:118, y:1341 and h:41px, w:19px.
[NOTE] Upper handle stats: x:869, y:1341 and h:41px, w:30px.
[WARN] Threw and caught a StaleElementReferenceException
See anything?
Traceback (most recent call last):
File "whoscored-scrape.py", line 442, in <module>
print("[NOTE]\t\tLower handle stats: x:{}, y:{} and h:{}px, w:{}px.".format( tlh.location['x'], tlh.location['y'], tlh.size['height'], tlh.size['width'] ))
File "/usr/local/Cellar/python3/3.6.2/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/selenium/webdriver/remote/webelement.py", line 404, in location
old_loc = self._execute(Command.GET_ELEMENT_LOCATION)['value']
File "/usr/local/Cellar/python3/3.6.2/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/selenium/webdriver/remote/webelement.py", line 493, in _execute
return self._parent.execute(command, params)
File "/usr/local/Cellar/python3/3.6.2/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/selenium/webdriver/remote/webdriver.py", line 256, in execute
self.error_handler.check_response(response)
File "/usr/local/Cellar/python3/3.6.2/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/selenium/webdriver/remote/errorhandler.py", line 194, in check_response
raise exception_class(message, screen, stacktrace)
selenium.common.exceptions.StaleElementReferenceException: Message: stale element reference: element is not attached to the page document
(Session info: chrome=65.0.3325.181)(Driver info: chromedriver=2.36.540469 (1881fd7f8641508feb5166b7cae561d87723cfa8),platform=Mac OS X 10.12.6 x86_64)
Supporting imagery:
