WebDriverWait works but page_source still returns half rendered HTML

Question

I have read Wait Until Page is Loaded, How to use Selenium Wait, Explicit Wait and other documentations to wait for a page to load and then start scraping. The wait successfully passes but I still get the same half/incomplete rendered HTML code.

from selenium import webdriver
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.common.by import By
from selenium.common.exceptions import TimeoutException

# prepare the option for the chrome driver
options = webdriver.ChromeOptions()
options.add_argument('headless')

# start chrome browser
browser = webdriver.Chrome(options=options,executable_path='C:/chromedriver_win32/chromedriver.exe')
browser.get('https://swappa.com/listing/view/LTNZ94446')

try:
    WebDriverWait(browser, 30).until(EC.presence_of_element_located((By.ID, "wrap")))
    print(browser.page_source)
except TimeoutException:
    print("not found")

For this my output starts somewhere half-way rather than from <html> at the top.

<div class="col-xs-6 col-sm-2 col-md-2">
                <div class="img-container" style="margin-bottom: 15px;">


                        <a href="https://static.swappa.com/media/listing/LTNZ94446/mhhHypyw.jpg" class="lightbox">
                            <img class="img-responsive" src="https://static.swappa.com/images/cache/7b/67/7b679a1d89816bc341a802f19f661eac.jpg" alt="Listing Image" style="margin:0px 0px 0px 0px; ">
                        </a>




                </div>
            </div>

I am not sure where is it going wrong.

It is clearly able to see the presence of element ID. (<div id="wrap">) since it doesnt throw timeout error
I tried using visibility of element, still no luck
Tried using readystate as well but no luck.

If there are ways using other libraries such as BeautifulSoup/URLLib/URLlib2/Scrapy, those would be helpful as well

I found a similar question that might help https://stackoverflow.com/a/30385843/8692977. Also, I would highly recommend trying requests_html it's relatively new, very easy to use and powerful https://pypi.org/project/requests-html. This library would replace bs4, requests, and selenium (to render js) — Marsilinou Zaky, Dec 30 '19 at 08:41
I've mentioned in my post that I've used this exact solution and it hasn't worked out. I'll give requests_html a try — Pirate X, Dec 30 '19 at 08:48

score 1 · Answer 1 · answered Dec 30 '19 at 09:13

You can check if page fully loaded using JavaScript:

options = webdriver.ChromeOptions()
options.add_argument('headless')

# start chrome browser
browser = webdriver.Chrome(options=options)
browser.get('https://swappa.com/listing/view/LTNZ94446')

WebDriverWait(browser, 30).until(lambda d: d.execute_script(
        'return ["complete", "interactive"].indexOf(document.readyState) != -1'))

# or use only complete
# WebDriverWait(browser, 30).until(lambda d: d.execute_script('return document.readyState == "complete"'))
print(browser.page_source)

score 0 · Answer 2 · answered Dec 30 '19 at 09:17

0

You can use python requests module.

Code:

import requests
response=requests.get("https://swappa.com/listing/view/LTNZ94446")
if response.status_code==200:
    print(response.text)

answered Dec 30 '19 at 09:17

KunduK

32,888
5
17
41

WebDriverWait works but page_source still returns half rendered HTML

2 Answers2