0

So far I've looked at this, this.

I'm trying to parse an HTML source using Selenium. To make things easier (or so I thought), I took the HTML from the web page I'm going to parse and put it into a local HTML file.

BeautifulSoup has no problem viewing the HTML, but Selenium just doesn't see it for some reason.

HTML:

<html><head>
<meta http-equiv="Cache-control" content="no-cache">
<title>NOTICE TO CORES USERS</title>
</head>
<body>
<center>
<b>

<h1>Welcome</h1>
Hours of Operation<br><br>
Monday-Friday 6:00am - 10:00pm<br>
Saturday 8:00am - 6:00pm<br>
Sunday 12:00 noon - 6:00pm<br>
<br>
<br>
<h4><font color="purple"><p><b><u>CORES ORACLE UPGRADE</u></b><br>
<br>
<font color="BLACK">
<font color="RED">Due to a recent technical upgrade, CORES is experiencing a number of issues. We are aware of these issues and our teams are working to resolve them. Corporate Registry will provide updates when available. Corporate Registry apologizes for any inconvenience.
<br>
<br>
<font color="RED">Effective February 3, 2019, <font color="BLACK"> Corporate Registry will send annual return reminders by email to corporations, non-profit organizations, limited liability partnerships, and cooperatives where there is an email address on record.<br>
<br>
Annual return reminders will be emailed about two weeks before the annual return is due.  The reminders will continue to be sent by regular mail
when there is no e-mail address on file or when there is a notice because the previous year's annual return has not been filed.  Directors of Alberta corporations will continue to receive copies of the outstanding annual return notice by regular mail.
<br>
<br>
</font></font></font></font></p><h4><font color="BLACK"><font color="RED"><font color="RED"><font color="BLACK"><font color="purple"><p><b><u>EXTENDED OUTAGE DATES</u></b><br>
<font color="RED"></font></p><p align="CENTRE"><font color="RED">
FULL DAY outages to allow for technical preventive maintenance are as follows:
<br>
<br>
<font color="BLACK">Sunday, May 12, 2019<br>
<font color="BLACK">Sunday, June 9, 2019<br>
<font color="BLACK">Sunday, July 14, 2019<br>
<font color="BLACK">Sunday, August 11, 2019<br>
<font color="BLACK">Sunday, September 8, 2019<br>
<font color="BLACK">Sunday, October 13, 2019<br>
<font color="BLACK">Sunday, November 10, 2019<br>
<font color="BLACK">Sunday, December 8, 2019<br>

</font></font></font></font></font></font></font></font></font></p><p align="CENTRE"></p><h5><font color="RED"><font color="BLACK"><font color="BLACK"><font color="BLACK"><font color="PURPLE">Updated: April 30, 2019</font></font></font></font></font></h5><p></p><font color="RED"><font color="BLACK"><font color="BLACK"><font color="BLACK"><font color="PURPLE">

<br>
<br>
<form action="cr_login.menu_frame" method="post">
<input type="hidden" name="p_default_menu" value="5">
<input type="hidden" name="p_system" value="CR">
<input type="hidden" name="p_accreditation" value="1">
<input type="hidden" name="p_spuid" value="30825">
<input type="hidden" name="p_userid" value="A02526">
<input type="submit" value="Continue">
</form>
</font></font></font></font></font></font></font></font></font></font></h4></font></h4></b></center><b><font color="purple"><font color="RED"><font color="RED"><font color="purple"><font color="RED"><font color="BLACK"><font color="BLACK"><font color="BLACK"><font color="PURPLE">


</font></font></font></font></font></font></font></font></font></b></body></html>

Python:

from bs4 import BeautifulSoup

from selenium import webdriver
from selenium.webdriver.chrome.options import Options
from selenium.webdriver.common.keys import Keys

chrome_options = Options()
chrome_options.binary_location = '/usr/bin/google-chrome'
chrome_options.add_argument('--headless')
chrome_options.add_argument('--no-sandbox')
chrome_options.add_argument('--disable-dev-shm-usage')

driver = webdriver.Chrome(chrome_options=chrome_options)
#driver.get('file:/c/Users/lanes/learning/product-repo/backend/functions/src/cores-scraper/oracle_upgrade.html')
driver.implicitly_wait(5)
driver.get('file:oracle_upgrade.html')

print('page source:', driver.page_source)

soup = BeautifulSoup(open('oracle_upgrade.html', 'r'), 'html.parser')

print('\nsoup:', soup)
    
driver.close()

Output:

page source: <html><head></head><body></body></html>

soup: <html><head>
<meta content="no-cache" http-equiv="Cache-control"/>
<title>NOTICE TO CORES USERS</title>
</head>
<body>
<center>
<b>
<h1>Welcome</h1>
Hours of Operation<br/><br/>
Monday-Friday 6:00am - 10:00pm<br/>
Saturday 8:00am - 6:00pm<br/>
Sunday 12:00 noon - 6:00pm<br/>
<br/>
<br/>
<h4><font color="purple"><p><b><u>CORES ORACLE UPGRADE</u></b><br/>
<br/>
<font color="BLACK">
<font color="RED">Due to a recent technical upgrade, CORES is experiencing a number of issues. We are aware of these issues and our teams are working to resolve them. Corporate Registry
will provide updates when available. Corporate Registry apologizes for any inconvenience.
<br/>
<br/>
<font color="RED">Effective February 3, 2019, <font color="BLACK"> Corporate Registry will send annual return reminders by email to corporations, non-profit organizations, limited liability partnerships, and cooperatives where there is an email address on record.<br/>
<br/>
Annual return reminders will be emailed about two weeks before the annual return is due.  The reminders will continue to be sent by regular mail
when there is no e-mail address on file or when there is a notice because the previous year's annual return has not been filed.  Directors of Alberta corporations will continue to receive copies of the outstanding annual return notice by regular mail.
<br/>
<br/>
</font></font></font></font></p><h4><font color="BLACK"><font color="RED"><font color="RED"><font color="BLACK"><font color="purple"><p><b><u>EXTENDED OUTAGE DATES</u></b><br/>
<font color="RED"></font></p><p align="CENTRE"><font color="RED">
FULL DAY outages to allow for technical preventive maintenance are as follows:
<br/>
<br/>
<font color="BLACK">Sunday, May 12, 2019<br/>
<font color="BLACK">Sunday, June 9, 2019<br/>
<font color="BLACK">Sunday, July 14, 2019<br/>
<font color="BLACK">Sunday, August 11, 2019<br/>
<font color="BLACK">Sunday, September 8, 2019<br/>
<font color="BLACK">Sunday, October 13, 2019<br/>
<font color="BLACK">Sunday, November 10, 2019<br/>
<font color="BLACK">Sunday, December 8, 2019<br/>
</font></font></font></font></font></font></font></font></font></p><p align="CENTRE"></p><h5><font color="RED"><font color="BLACK"><font color="BLACK"><font color="BLACK"><font color="PURPLE">Updated: April 30, 2019</font></font></font></font></font></h5><p></p><font color="RED"><font color="BLACK"><font color="BLACK"><font color="BLACK"><font color="PURPLE">
<br/>
<br/>
<form action="cr_login.menu_frame" method="post">
<input name="p_default_menu" type="hidden" value="5"/>
<input name="p_system" type="hidden" value="CR"/>
<input name="p_accreditation" type="hidden" value="1"/>
<input name="p_spuid" type="hidden" value="30825"/>
<input name="p_userid" type="hidden" value="A02526"/>
<input type="submit" value="Continue"/>
</form>
</font></font></font></font></font></font></font></font></font></font></h4></font></h4></b></center><b><font color="purple"><font color="RED"><font color="RED"><font color="purple"><font color="RED"><font color="BLACK"><font color="BLACK"><font color="BLACK"><font color="PURPLE">
</font></font></font></font></font></font></font></font></font></b></body></html>

Question:

Why does Selenium not "see" the HTML in the body like Soup does?

Community
  • 1
  • 1
ChumiestBucket
  • 868
  • 4
  • 22
  • 51
  • 1
    Can you load the page in non-headless mode to make sure the data is loaded when try to open the page. I guess `driver.get('file:oracle_upgrade.html')` is not loading the page completely. – supputuri Jun 05 '19 at 15:20
  • @supputuri running the code without `--headless` arg results in the same error, not sure why. – ChumiestBucket Jun 05 '19 at 15:52

2 Answers2

2

I was able to replicate your issue.

You are getting tricked because driver.get() wont really return an error if it is not able to load your file. Instead the driver.page_source will contain an almost empty document. I am not sure where your file is located, but i think you are just missing that a file URI should start with file:// and not file:

The following code worked for me:

from bs4 import BeautifulSoup
from selenium import webdriver
from selenium.webdriver.chrome.options import Options
from selenium.webdriver.common.keys import Keys

chrome_options = Options()
chrome_options.add_argument('--headless')
chrome_options.add_argument('--no-sandbox')
chrome_options.add_argument('--disable-dev-shm-usage')

driver = webdriver.Chrome(chrome_options=chrome_options)
driver.get('file:///Users/jimmy/src/stackoverflow/html-elements-missing-from-selenium-page-source-but-can-be-found-using-beautifu/oracle_upgrade.html')
print('page source:', driver.page_source)
driver.close()

You should not expect the exactly same output as the input since chrome will "fix" your html for you. For example if you forget <html> it will politely add it to the source for you without complaining.

Jimmy Engelbrecht
  • 653
  • 6
  • 9
  • 1
    I tried adjusting the file path as mentioned, so it looks like this: `'file:///c/Users/path/to/the/file.html` and it was able to read the html file as expected. who knew it was something so simple? the three `/`'s was the key. – ChumiestBucket Jun 05 '19 at 15:57
  • 1
    That's the reason why I always prefer checking the page in non-headless mode first, before head-less. – supputuri Jun 05 '19 at 16:01
1

I cannot reproduce your issue using:

  • Python 3.7.3
  • Selenium 3.141.0 (according to pip show command
  • Chromedriver 74.0.3729.6
  • Chrome 74.0.3729.169

enter image description here

So I would recommend

  • upgrading to the latest selenium package version using pip like:

    pip install --upgrade selenium

  • cross-check your Chrome and ChromeDriver versions - it must be 100% match

  • alternatively you can try getting the page source via XPath expression 

    print(driver.find_element_by_xpath("/html").get_attribute("innerHTML"))

    enter image description here

Dmitri T
  • 159,985
  • 5
  • 83
  • 133