0

I am trying to scrape all of the available odds for each game found on this webpage: https://www.sportsbookreview.com/betting-odds/nfl-football/?date=20170917

I know the webpage is loaded dynamically, so I attempted to insert a scroll hoping that it would load all of the available odds as it scrolls, but that unfortunately does not seem to be the case as it just removes the previously loaded data as it continues to scroll.

I have tried implementing similar posts that have this issue, (like this Trying to use Python and Selenium to scroll and scrape a webpage iteratively), but I still just can't seem to solve it. Pasted below is my code. 

import selenium
from selenium import webdriver

url= 'https://www.sportsbookreview.com/betting-odds/nfl-football/?date=20170917'
driver = webdriver.Chrome()
driver.get(url)
driver.execute_script("window.scrollTo(0, 900)") 

odds_finder=driver.find_elements_by_class_name('_3h0tU')

file_odds = []
for x in odds_finder:
    x=x.text
    file_odds.append(x)

driver.quit()

The output of file_odds is pasted below, but as you can tell the first very elements are only the "opener" and "wagers" rows of the games and not the remainder of the available odds like is scraped later in the list. Any help on this would be greatly appreciated.

['64%',
 '36%',
 'PK-110',
 'PK-110',
 '56%',
 '44%',
 '+7½-110',
 '-9+105',
 '58%',
 '42%',
 '+7-105',
 '-7-115',
 '66%',
 '34%',
 '-4½-110',
 '+4½-110',
 '49%',
 '51%',
 '-7-110',
 '+7-110',
 '45%',
 '55%',
 '+4½-110',
 '-4½-110',
 '49%',
 '51%',
 '+7½-130',
 '-7½+110',
 '+8½-104',
 '-8½-106',
 '+8½-105',
 '-8½-105',
 '+8-110',
 '-8-110',
 '+8½-110',
 '-8½-110',
 '+9-110',
 '-9-110',
 '+8½-105',
 '-8½-105',
 '53%',
 '47%',
 '+6-110',
 '-6-110',
 '+7-100',
 '-7-110',
 '+7-105',
 '-7-105',
 '+7-119',
 '-7-101',
 '+7-110',
 '-7-110',
 '+7-110',
 '-7-110',
 '+6½+105',
 '-6½-115',
 '49%',
 '51%',
 '+4-110',
 '-4-110',
 '+3½-105',
 '-3½-105',
 '+3½-105',
 '-3½-105',
 '+3½-110',
 '-3½-110',
 '+3½-110',
 '-3½-110',
 '+3½-110',
 '-3½-110',
 '+3½-110',
 '-3½+100',
 '37%',
 '63%',
 '+14½-120',
 '-14½+100',
 '+14-100',
 '-14-110',
 '+14-105',
 '-14-105',
 '+14-114',
 '-14-106',
 '+14-110',
 '-14-110',
 '+14+100',
 '-14-120',
 '+13½+105',
 '-13½-115',
 '53%',
 '47%',
 '+3-120',
 '-3+100',
 '+3-106',
 '-3-104',
 '+3-110',
 '-3+100',
 '+3-112',
 '-3-108',
 '+3-110',
 '-3-110',
 '+3-105',
 '-3-115',
 '+3-105',
 '-3-105',
 '60%',
 '40%',
 '-1-120',
 '+1+100',
 '-2½-100',
 '+2½-110',
 '-2½-103',
 '+2½-107',
 '-2½-105',
 '+2½-115',
 '-2½-118',
 '+2½-102',
 '-3-105',
 '+3-115',
 '-2½-105',
 '+2½-105',
 '41%',
 '59%',
 '+14-130',
 '-14+110',
 '+13½-110',
 '-13½-100',
 '+13½-108',
 '-13½-102',
 '+13½-115',
 '-13½-105',
 '+13½-110',
 '-13½-110',
 '+14-105',
 '-14-115',
 '+13½-105',
 '-13½-105',
 '51%',
 '49%',
 '+2½+100',
 '-2½-120',
 '+3½-110',
 '-3½-100',
 '+3+108',
 '-3-118',
 '+3+105',
 '-3-125',
 '+3+110',
 '-3-130',
 '+3-105',
 '-3-115',
 '+3+110',
 '-3-120']
Cam
  • 105
  • 1
  • 10

1 Answers1

0

try with:

//div[contains(@class,'_3A-gC')]//section//div[starts-with(@class, '_3h0tU')]

Infern0
  • 2,565
  • 1
  • 8
  • 21
  • thank you, I am trying to use this but am getting an error saying compound classes are not allowed, would you mind showing me how I would include this in my original script? – Cam Oct 07 '18 at 19:18
  • actually sorry I realized that this was an Xpath I think? I tried it as that and it still did not give me the full odds, unless I am missing something – Cam Oct 07 '18 at 19:40
  • can you specify which content is missing ? – Infern0 Oct 08 '18 at 06:22