lxml xpath unable to display html items

Question

I'm trying to use lxml to parse a webpage below. But something seems to be wrong with my xpath. I'm not sure what am I doing wrong.

web_content = requests.get(r"https://www.quandl.com/data/TSE").content
dataset_count = html.fromstring(web_content)
print(dataset_count.xpath(r'//*[@id="ember667"]/div[2]/main/section/section/section[2]/div[3]/div[2]/span[2]'))

I'm trying to get it to return the dataset number of 3908. But this xpath doesn't seem to work for me. Any thoughts?

Also, I'm hoping that if I pass another quandl link through requests, I can use the same xpath to draw out the dataset number. Would that be possible?

`3908` is not really part of the HTML source you get with this method. — Mathias Müller, Feb 17 '16 at 14:30

paul trmbrth · Accepted Answer · 2016-02-24T09:50:44.867

It seems the datasets count is also in a <noscript> element:

<div class='centered' id='main' role='main'>
<div id='content'>
<noscript>
<table>
<tbody>
<tr>
<td>Database Name</td>
<td>Tokyo Stock Exchange</td>
</tr>
<tr>
<td></td>
<td></td>
</tr>
<tr>
<td>Datasets</td>
<td>3908</td>
</tr>
<tr>
<td>Downloads</td>
<td>4067259</td>
</tr>
<tr>
...

So you can grab that using something like this:

>>> import requests
>>> import lxml.html

>>> r = requests.get('https://www.quandl.com/data/TSE')
>>> h = lxml.html.fromstring(r.text)
>>> h
<Element html at 0x7ffb5f6ed0a8>

>>> h.xpath('//noscript')
[<Element noscript at 0x7ffb5c16ac58>, <Element noscript at 0x7ffb5c16ac00>]

>>> h.xpath('string(//noscript//tr[td[1]="Datasets"]/td[2])')
'3908'
>>> h.xpath('string(//div[@id="content"]//noscript//tr[td[1]="Datasets"]/td[2])')
'3908'
>>> h.xpath('number(//div[@id="content"]//noscript//tr[td[1]="Datasets"]/td[2])')
3908.0

Explanation on the XPath as requested by OP:

//div[@id="content"]          <-- look for a <div> element with "id" attribute equal to "content"
  //noscript                  <-- look for a <noscript> descendant
    //tr[                     <-- look for a <tr> descendant...
        td[1]="Datasets"      <-- ... which 1st <td> child string value is "Datasets"...
                              (this is true if the <td> contains only 1 text node "Datasets"
        ]
      /td[2]                  <-- select the 2nd <td> of previous matching <tr> rows

Thanks. Was wondering if you could give a brief explanation about the `tr td` code you wrote? if its not too much trouble? — jake wong, Feb 20 '16 at 14:15
@jakewong, I added an explanation on the XPath expression parts. — paul trmbrth, Feb 24 '16 at 09:51

score 0 · Answer 2 · answered Feb 17 '16 at 14:30

There is no 3908 number in the response that requests receives since the number is loaded dynamically with an additional request.

One option to solve it is to use a real browser and control it with selenium. Here is an example working code that uses PhantomJS headless browser:

from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC


driver = webdriver.PhantomJS()
driver.get("https://www.quandl.com/data/TSE")

wait = WebDriverWait(driver, 10)
elm = wait.until(EC.presence_of_element_located((By.CSS_SELECTOR, ".database-statistics .column:nth-child(2) span:nth-child(2)")))
print(elm.text)

driver.close()

Prints 3,908.

lxml xpath unable to display html items

2 Answers2

Linked