Extract data from HTML table using Python

Question

I want to extract data from HTML table using Python script and save it as variables(that I can later use in same script after loading them in if they exist) into a separate file. Also I want the script to ignore the first row of table(Component, Status, Time / Error). I would prefer not to use external libraries.

The output into a new file should be like so:

SAVE_DOCUMENT_STATUS = "OK"
SAVE_DOCUMENT_TIME = "0.408"
GET_DOCUMENT_STATUS = "OK"
GET_DOCUMENT_TIME = "0.361"
...

And heres the input to the script:

<table border=1>
<tr>
<td><b>Component</b></td>
<td><b>Status</b></td>
<td><b>Time / Error</b></td>
</tr>
<tr><td>SAVE_DOCUMENT</td><td>OK</td><td>0.408 s</td></tr>
<tr><td>GET_DOCUMENT</td><td>OK</td><td>0.361 s</td></tr>
<tr><td>DVK_SEND</td><td>OK</td><td>0.002 s</td></tr>
<tr><td>DVK_RECEIVE</td><td>OK</td><td>0.002 s</td></tr>
<tr><td>GET_USER_INFO</td><td>OK</td><td>0.135 s</td></tr>
<tr><td>NOTIFICATIONS</td><td>OK</td><td>0.002 s</td></tr>
<tr><td>ERROR_LOG</td><td>OK</td><td>0.001 s</td></tr>
<tr><td>SUMMARY_STATUS</td><td>OK</td><td>0.913 s</td></tr>
</table>

I tried to do it in bash, but since I need to compare *_TIME variables to maximum time, then it fails, because they're float numbers.

"One more thing, I don't want to use external libraries". Epic Fail. You really must use Beautiful Soup. It's the best for this kind of thing. — S.Lott, Jul 29 '11 at 19:25
Ok, if there aren't any other way out I will use Beautiful Soup then:) — Marko, Jul 29 '11 at 19:27

score 4 · Accepted Answer · answered Jul 29 '11 at 19:46

Using lxml:

import lxml.html as lh

content='''\
<table border=1>
<tr>
<td><b>Component</b></td>
<td><b>Status</b></td>
<td><b>Time / Error</b></td>
</tr>
<tr><td>SAVE_DOCUMENT</td><td>OK</td><td>0.408 s</td></tr>
<tr><td>GET_DOCUMENT</td><td>OK</td><td>0.361 s</td></tr>
<tr><td>DVK_SEND</td><td>OK</td><td>0.002 s</td></tr>
<tr><td>DVK_RECEIVE</td><td>OK</td><td>0.002 s</td></tr>
<tr><td>GET_USER_INFO</td><td>OK</td><td>0.135 s</td></tr>
<tr><td>NOTIFICATIONS</td><td>OK</td><td>0.002 s</td></tr>
<tr><td>ERROR_LOG</td><td>OK</td><td>0.001 s</td></tr>
<tr><td>SUMMARY_STATUS</td><td>OK</td><td>0.913 s</td></tr>
</table>
'''
tree=lh.fromstring(content)
for key, status, t in zip(*[iter(tree.xpath('//td/text()'))]*3):
    print('''{k}_STATUS = "{s}"
{k}_TIME = "{t}"'''.format(k=key,s=status,t=t.rstrip(' s')))

yields

SAVE_DOCUMENT_STATUS = "OK"
SAVE_DOCUMENT_TIME = "0.408"
GET_DOCUMENT_STATUS = "OK"
GET_DOCUMENT_TIME = "0.361"
DVK_SEND_STATUS = "OK"
DVK_SEND_TIME = "0.002"
DVK_RECEIVE_STATUS = "OK"
DVK_RECEIVE_TIME = "0.002"
GET_USER_INFO_STATUS = "OK"
GET_USER_INFO_TIME = "0.135"
NOTIFICATIONS_STATUS = "OK"
NOTIFICATIONS_TIME = "0.002"
ERROR_LOG_STATUS = "OK"
ERROR_LOG_TIME = "0.001"
SUMMARY_STATUS_STATUS = "OK"
SUMMARY_STATUS_TIME = "0.913"

lxml is an external library, no? Depending on @Marko Python version, he ca use [ElementTree](http://docs.python.org/library/xml.etree.elementtree.html) instead, or - urgh.... - [xml.dom.minidom](http://docs.python.org/library/xml.dom.minidom.html) — brandizzi, Jul 29 '11 at 20:06
@brandizzi: In the comments Marko expressed a willingness to use BeautifulSoup. I took that to mean a willingness to use any external library. — unutbu, Jul 29 '11 at 20:12
Here's my final code thanks to unutbu, I still used the external library: `if (not os.path.exists(Filename)): download = urllib2.urlopen(MonitorURL) data = download.read() fileObj = open(Filename, "w") tree = lh.fromstring(data) for key, status, time in zip(*[iter(tree.xpath('//td/text()'))]*3): fileObj.writelines('''{k}_STATUS = "{s}"\n'''.format(k=key,s=status)) fileObj.writelines('''{k}_TIME = "{t}"\n'''.format(k=key,t=time.rstrip(' s'))) fileObj.close()` — Marko, Jul 29 '11 at 20:19

score 2 · Answer 2 · answered Jul 29 '11 at 19:41

Well, if your HTML document really has such a stable structure (which makes me scratch my head because it is pretty rare) you can use regexes:

>>> import re
>>> r = re.compile('<tr><td>(.*)</td><td>(.*)</td><td>(.*) s</td></tr>')

The regex below groups the values you want to show in the result. Then you use the sub() method of the object. If the text is in a variable (such as content) just execute it this way:

r.sub(r'\1_STATUS = "\2"\n\1_TIME = \3', content)

The result:

>>> print r.sub(r'\1_STATUS = "\2"\n\1_TIME = \3', content)
<table border=1>
<tr>
<td><b>Component</b></td>
<td><b>Status</b></td>
<td><b>Time / Error</b></td>
</tr>
SAVE_DOCUMENT_STATUS = "OK"
SAVE_DOCUMENT_TIME = 0.408
GET_DOCUMENT_STATUS = "OK"
GET_DOCUMENT_TIME = 0.361
DVK_SEND_STATUS = "OK"
DVK_SEND_TIME = 0.002
DVK_RECEIVE_STATUS = "OK"
DVK_RECEIVE_TIME = 0.002
GET_USER_INFO_STATUS = "OK"
GET_USER_INFO_TIME = 0.135
NOTIFICATIONS_STATUS = "OK"
NOTIFICATIONS_TIME = 0.002
ERROR_LOG_STATUS = "OK"
ERROR_LOG_TIME = 0.001
SUMMARY_STATUS_STATUS = "OK"
SUMMARY_STATUS_TIME = 0.913
</table>

Sure, there is a lot of garbage in the string yet, but it gives the idea :)

If your HTML documents are not that stable, however, you should really consider some XML parser or, better yet, BeautifulSoup, because it would be a heck of a job to process an unstably structured HTML file by hand.

-1 http://stackoverflow.com/questions/1732348/regex-match-open-tags-except-xhtml-self-contained-tags/1732454#1732454 — SingleNegationElimination, Jul 29 '11 at 19:48
The HTML I gave is stable :) The script is needed for making this old HTML edible to Nagios. — Marko, Jul 29 '11 at 19:53
@TokenMacGuy although I agree in general, I am with [this answer](http://stackoverflow.com/questions/1732348/regex-match-open-tags-except-xhtml-self-contained-tags/1733489#1733489) in this case :) — brandizzi, Jul 29 '11 at 20:01

Extract data from HTML table using Python

2 Answers2