1

So I'm writing a Python script that checks Blackboard (school interface site) for updates. But the HTML I receive back from my script is not completely the same as the HTML when viewed in my browser. I'm unsure if this is a cookie issue or what I'm missing.

USERNAME = ''
PASSWORD = ''

updates = 0  
site = 'http://schoolsite.edu'

browser = mechanize.Browser()
browser.open(site)
browser.select_form(nr = 0)
browser.form['j_username'] = USERNAME
browser.form['j_password'] = PASSWORD
browser.submit()

#it brings back an empty form, just submit it.
browser.select_form(nr = 0)
browser.submit()

html_resp = browser.response().read()

The HTML in question looks like this (this is from the script)

<span id="badgeTotal" style="visibility: hidden" title="">
<span class="hideoff" id="badgeAXLabel">Activity Updates</span>
<span class="badge" id="badgeTotalCount" title=""></span>

What I was expecting it to look like (from Chrome/actual browser)

<span id="badgeTotal" style="visibility: visible;" title="">
<span class="hideoff" id="badgeAXLabel">Activity Updates</span>
<span class="badge" id="badgeTotalCount" title="">1</span>

What I'm really after is that '1' number in the last line, but I feel like the visibility attribute is holding it back. Note that I get the same cookies from Mechanize that I also get in browser. (not exact same, but same id, name, ect.)

Any thoughts?

Any input is appreciated.

alecxe
  • 462,703
  • 120
  • 1,088
  • 1,195
radar
  • 13
  • 3

1 Answers1

0

Pretty sure there is javascript involved which mechanize cannot handle.

An alternative solution here would be to automate a real browser through selenium:

from selenium import webdriver
from selenium.common.exceptions import TimeoutException
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC

driver = webdriver.Firefox()  # could also be headless: webdriver.PhantomJS()
driver.get('http://schoolsite.edu')

# submit a login form
username = driver.find_element_by_name('j_username')
password = driver.find_element_by_name('j_password')

username.send_keys(USERNAME)
password.send_keys(PASSWORD)

username.submit()

# wait for the badge count to appear
badge_count = WebDriverWait(driver, 10).until(EC.presence_of_element_located((By.ID, "badgeTotalCount")))

print(badge_count.text)
Community
  • 1
  • 1
alecxe
  • 462,703
  • 120
  • 1,088
  • 1,195
  • Wow! Awesome. Thanks @alecxe. It worked! I didn't want to deal with Firefox opening and closing, so I used this [post](http://stackoverflow.com/questions/5370762/how-to-hide-firefox-window-firefox-webdriver). It was good thing you added that comment about PhantomJS. I also had to add a delay for it to grab the badge count. So the last couple lines look like: `badge_count = WebDriverWait(driver, 10).until(EC.presence_of_element_located((By.ID, "badgeTotalCount")))` `sleep(2)` `print(badge_count.text)` I know that's not elegant, but it works. – radar Mar 08 '15 at 05:59