Selenium loading, but not printing all HTML

Question

I am attempting to use Python and Selenium to web-scrape dynamically loaded data from a website. The problem is, only about half of the data is being reported as present, when in reality it all should be there. Even after using pauses before printing out all the page content, or simple find element by class searches, there seems to be no solution. The URL of the site is https://www.sportsbookreview.com/betting-odds/nfl-football/consensus/?date=20180909. As you can see, there are 13 main sections, however I am only able to retrieve data from the first four games. To best show the problem I'll attach the code for printing the inner-HTML for the entire page to show the discrepancies between the loaded and non-loaded data.

from selenium import webdriver
import requests

url = "https://www.sportsbookreview.com/betting-odds/nfl-football/consensus/?date=20180909"
driver = webdriver.Chrome()
driver.get(url)
print(driver.execute_script("return document.documentElement.innerText;"))

EDIT: The problem is not the wait time, for I am running it line by line and fully waiting for it to load. It appears the problem boild down to selenium not grabbing all the JS loaded text on the page, as seen by the console output in the answer below.

Are you sure you don't want `document.documentElement.innerHTML` ? or maybe `document.body.innerHTML` ? — pguardiario, Nov 12 '18 at 06:01
@pguardiario Why are you posting to exact duplicate comments? — U13-Forward, Nov 12 '18 at 06:03
Those are actually slightly different. I recommend using `document.body.innerHTML` — pguardiario, Nov 12 '18 at 11:09

sudonym · Answer 1 · 2018-11-13T01:34:20.083

1

This solution is only worth to consider if there are lots of WebDriverWait calls and given the interest in reduced runtime - else go for DebanjanB's approach

You need to wait some time to let your html load completely. Also, you can set a timeout for script execution. To add a unconditional wait to driver.get(URL) in selenium, driver.set_page_load_timeout(n) with n = time/seconds and loop:

driver.set_page_load_timeout(n)         # Set timeout of n seconds for page load
loading_finished = 0                    # Set flag to 0
while loading_finished == 0:            # Repeat while flag = 0
    try:
       sleep(random.uniform(0.1, 0.5))  # wait some time
       website = driver.get(URL)        # try to load for n seconds
       loading_finished = 1             # Set flag to 1 and exit while loop
       logger.info("website loaded")    # Indicate load success
    except:
       logger.warn("timeout - retry")   # Indicate load fail
else:                                   # If flag == 1
    driver.set_script_timeout(n)        # Set timeout of n seconds for script  
    script_finished = 0                 # Set flag to 0
    while script_finished == 0          # Second loop
       try:
          print driver.execute_script("return document.documentElement.innerText;")       
          script_finished = 1           # Set flag to 1
          logger.info("script done")    # Indicate script done
       except:                          
          logger.warn("script timeout") 
    else:
        logger.info("if you're still missing html here, increase timeout")

edited Nov 13 '18 at 01:34

answered Nov 12 '18 at 06:01

sudonym

3,788
4
36
61

use WebDriverWait, not this. – Corey Goldberg Nov 12 '18 at 13:17
Not sure if you noticed though, but the problem is still present in the console output. But for the first 4 games, all of the wager data is retrieved, however for the next 9 games, only the team names and game start times are return, when looking at the actual page, all elements are present and loaded by the JS. – Kyle Lagerquist Nov 12 '18 at 20:15
taking into account that you are executing a script, you could `driver.set_script_timeout(n)` [link](https://stackoverflow.com/questions/48941260) - I have updated my answer – sudonym Nov 13 '18 at 01:06
Great thought, but still not returning the same data for each div. – Kyle Lagerquist Nov 13 '18 at 01:14
admittedly, my attempt now looks length and ugly - did you try the solution of @DebanjanB – sudonym Nov 13 '18 at 01:17
Yes I did, but look at the difference in data retrieved in each block (there is a "/" separating each block). Although all the elements are present on the page, selenium is only "seeing" and being able to retrieve half of them. – Kyle Lagerquist Nov 13 '18 at 01:46

score 1 · Answer 2 · edited Nov 12 '18 at 13:18

@sudonym's analysis was in the right direction. You need to induce WebDriverWait for the desired elements to be visible before you attempt to extract them through execute_script() method as follows:

Code Block:

# -*- coding: UTF-8 -*-
from selenium import webdriver
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.common.by import By

url = "https://www.sportsbookreview.com/betting-odds/nfl-football/consensus/?date=20180909"
driver = webdriver.Chrome()
driver.get(url)
WebDriverWait(driver, 30).until(EC.visibility_of_all_elements_located((By.XPATH, "//h2[contains(.,'USA - National Football League')]//following::section//span[3]")))
print(driver.execute_script("return document.documentElement.innerText;"))

Console Output:

SPORTSBOOK REVIEW
Home
Best Sportsbooks
Rating Guide
Blacklist
Bonuses
BETTING ODDS
FREE PICKS
Sports Picks
NFL
College Football
NBA
NCAAB
MLB
NHL
More Sports
How to Bet
Tools
FORUM
Home
Players Talk
Sportsbooks & Industry
Newbie Forum
Handicapper Think Tank
David Malinsky's Point Blank
Service Plays
Bitcoin Sports Betting
NBA Betting
NFL Betting
NCAAF Betting
MLB Betting
NHL Betting
CONTESTS
EARN BETPOINTS
What Are Betpoints?
SBR Sportsbook
SBR Casino
SBR Racebook
SBR Poker
SBR Store
Today
NFL
NBA
NHL
MLB
College Football
NCAA Basketball
Soccer
Soccer Odds
Major League Soccer
UEFA Champions League
UEFA Nations League
UEFA Europa League
English Premier League
World Cup 2022
Tennis
Tennis Odds
ATP
WTA
UFC
Boxing
More Sports
CFL
WNBA
AFL
Betting Odds/NFL Odds/Consensus
TODAY
|
YESTERDAY
|
DATE
?
Login
?
Settings
?
Bet Tracker
?
Bet Card
?
Favorites
NFL Consensus for Sep 09, 2018
USA - National Football League
Sunday Sep 09, 2018
01:00 PM
/
Pittsburgh vs Cleveland
453
Pittsburgh
454
Cleveland
Current Line
-3½+105
+3½-115
Wagers Placed
10040
54.07%
8530
45.93%
Amount Wagered
$381,520.00
56.10%
$298,550.00
43.90%
Average Bet Size
$38.00
$35.00
SBR Contest Best Bets
22
9
01:00 PM
/
San Francisco vs Minnesota
455
San Francisco
456
Minnesota
Current Line
+6-102
-6-108
Wagers Placed
6250
41.25%
8900
58.75%
Amount Wagered
$175,000.00
29.50%
$418,300.00
70.50%
Average Bet Size
$28.00
$47.00
SBR Contest Best Bets
5
19
01:00 PM
/
Cincinnati vs Indianapolis
457
Cincinnati
458
Indianapolis
Current Line
-1-104
+1-106
Wagers Placed
11640
66.36%
5900
33.64%
Amount Wagered
$1,338,600.00
85.65%
$224,200.00
14.35%
Average Bet Size
$115.00
$38.00
SBR Contest Best Bets
23
12
01:00 PM
/
Buffalo vs Baltimore
459
Buffalo
460
Baltimore
Current Line
+7½-103
-7½-107
Wagers Placed
5220
33.83%
10210
66.17%
Amount Wagered
$78,300.00
16.79%
$387,980.00
83.21%
Average Bet Size
$15.00
$38.00
SBR Contest Best Bets
5
17
01:00 PM
/
Jacksonville vs N.Y. Giants
461
Jacksonville
462
N.Y. Giants
01:00 PM
/
Tampa Bay vs New Orleans
463
Tampa Bay
464
New Orleans
01:00 PM
/
Houston vs New England
465
Houston
466
New England
01:00 PM
/
Tennessee vs Miami
467
Tennessee
468
Miami
04:05 PM
/
Kansas City vs L.A. Chargers
469
Kansas City
470
L.A. Chargers
04:25 PM
/
Seattle vs Denver
471
Seattle
472
Denver
04:25 PM
/
Dallas vs Carolina
473
Dallas
474
Carolina
04:25 PM
/
Washington vs Arizona
475
Washington
476
Arizona
08:20 PM
/
Chicago vs Green Bay
477
Chicago
478
Green Bay
Media
Site Map
Terms of use
Contact Us
Privacy Policy
DMCA
18+. Gamble Responsibly.
© Sportsbook Review. All Rights Reserved.

actually this is the way to go for OP's setup. my reason to not do it this way was that for my use case, there have been so many `WebDriverWait.until()` calls that the execution time went off into infinity. I decided to just get `driver.page_source()` once and do the rest with BS/lxml. — sudonym, Nov 12 '18 at 08:31
Not sure if you noticed though, but the problem is still present in the console output. But for the first 4 games, all of the wager data is retrieved, however for the next 9 games, only the team names and game start times are return, when looking at the actual page, all elements are present and loaded by the JS. — Kyle Lagerquist, Nov 12 '18 at 18:52

Selenium loading, but not printing all HTML

2 Answers2