Having issues with Python xpath scraping

Question

I'm back again with a question for the wonderful people here :)

Ive recently begun getting back into python (50% done at codcademy lol) and decided to make a quick script for web-scraping the spot price of gold in CAD. This will eventually be a part of a much bigger script... but Im VERY rusty and thought it would be a good project.

My issue: I have been following the guide over at http://docs.python-guide.org/en/latest/scenarios/scrape/ to accomplish my goal, however my script always returns/prints

<Element html at 0xRANDOM>

with RANDOM being a (i assume) random hex number. This happens no matter what website I seem to use.

My Code:

#!/bin/python
#Scrape current gold spot price in CAD

from lxml import html
import requests

def scraped_price():
    page = requests.get('http://goldprice.org/gold-price-canada.html')
    tree = html.fromstring(page.content)

    print "The full page is: ", tree #added for debug WHERE ERROR OCCURS
    bid = tree.xpath("//span[@id='gpotickerLeftCAD_price']/text()")
    print "Scraped content: ", bid
    return bid
gold_scraper = scraped_price()

My research:

1) www.w3schools.com/xsl/xpath_syntax.asp

This is where I figured out to use '//span' to find all 'span' objects and then used the @id to narrow it down to the one I need.

2)Scraping web content using xpath won't work

This makes me think I simply have a bad tree.xpath setup. However I cannot seem to figure out where or why.

Any assistance would be greatly appreciated.

Answer 1

<Element html at 0xRANDOM>

What you see printed is the lxml.html's Element class string representation. If you want to see the actual HTML content, use tostring():

print(html.tostring(tree, pretty_print=True))

You are also getting Scraped content: [] printed which really means that there were no elements matching the locator. And, if you would see the previously printed out HTML, there is actually no element with id="gpotickerLeftCAD_price" in the downloaded source.

The prices on this particular site are retrieved dynamically with continuous JSONP GET requests issued periodically. You can either look into simulating these requests, or stay on a higher level automating a browser via selenium. Demo (using PhantomJS headless browser):

>>> import time
>>> from selenium import webdriver
>>> 
>>> driver = webdriver.PhantomJS()
>>> driver.get("http://goldprice.org/gold-price-canada.html")
>>> while True:
...     print(driver.find_element_by_id("gpotickerLeftCAD_price").text)
...     time.sleep(1)
... 
1,595.28
1,595.28
1,595.28
1,595.28
1,595.28
1,595.19
...