0

I am using Python bindings for Selenium with PhantomJS to scrape the contents of a website, like so.

The element I want to access is in the DOM but not in the HTML source. I understand that if I want to access elements in the DOM itself, I need to use the PhantomJS evaluate() function. (e.g. http://www.crmarsh.com/phantomjs/ ; http://phantomjs.org/quick-start.html)

How can I do this from within Selenium?

Here is part of my code (which is currently not able to access the element using a PhantomJS driver):

time.sleep(60)
driver.set_window_size(1024,768)
todays_points = driver.find_elements_by_xpath("//div/a[contains(text(),'Today')]/preceding-sibling::span")
total = 0
for today in todays_points: 
    driver.set_window_size(1024,768)
    points = today.find_elements_by_class_name("stream_total_points")[0].text
    points = int(points[:-4])
    total += points
driver.close()
print total
Community
  • 1
  • 1
lin_bug
  • 1,125
  • 11
  • 10
  • Are you sure that you're on the correct page? What's the output that you get and what is the output that you expect? You don't need `evaluate()` if you only want to access the element text. – Artjom B. Sep 17 '15 at 11:45
  • Yes, I am on the correct page (when I use print(driver.page_source) I get the page I expect). At the moment, my value for todays_points is an empty list (and ultimately, total = 0), whereas when I use chromedriver instead of PhantomJS I get a value back. – lin_bug Sep 17 '15 at 12:09
  • What PhantomJS version do you have and have you tried upgrading (currently v2.0.0)? – Artjom B. Sep 17 '15 at 12:15
  • I am using the most current version. – lin_bug Sep 17 '15 at 12:52

0 Answers0