Retrieving string from html on non-unique table

Question

Here is the html I am trying to parse.

<TD>Serial Number</TD><TD>AB12345678</TD>

I am attempting to use regex to parse the data. I heard about BeautifulSoup but there are around 50 items like this on the page all using the same table parameters and none of them have ID numbers. The closest they have to unique identifiers is the data in the cell before the data I need.

serialNumber = re.search("Serial Number</td><td>\n(.*?)</td>", source)

Source is simply the source code of the page grabbed using urllib. There is new line in the html between the second and the serial number but I am unsure if that matters.

Regex is traditionally an ill-advised solution to parsing HTML. You should really be using BeautifulSoup, descending into the table and grabbing data if and only if (condition). Else, go on to the next table. You could try [scrapy](http://scrapy.org/) and use it to write a spider, which normally incorporates something similar to a regex — inspectorG4dget, May 16 '11 at 19:27
Obligatory link: http://stackoverflow.com/questions/1732348/regex-match-open-tags-except-xhtml-self-contained-tags — Amadan, May 16 '11 at 19:32

score 2 · Answer 1 · answered May 17 '11 at 03:30

Pyparsing can give you a little more robust extractor for your data:

from pyparsing import makeHTMLTags, Word, alphanums

htmlfrag = """<blah></blah><TD>Serial Number</TD><TD>
            AB12345678
            </TD><stuff></stuff>"""

td,tdEnd = makeHTMLTags("td")

sernoFormat = (td + "Serial Number" + tdEnd + 
                td + Word(alphanums)('serialNumber') + tdEnd)


for sernoData in sernoFormat.searchString(htmlfrag):
    print sernoData.serialNumber

Prints:

AB12345678

Note that pyparsing doesn't care where the extra whitespace falls, and it also handles unexpected attributes that might crop up in the defined tags, whitespace inside tags, tags in upper/lower case, etc.

+1 for giving an example on how to use a parser and not only the usual "use a parser for html!!!" — stema, May 17 '11 at 07:49

score 0 · Answer 2 · answered May 17 '11 at 07:44

In most of the cases it is better to work on html using an appropriate parser, but for some cases it is perfectly OK to use regular expressions for the job. I do not know enough about your task to judge if it is a good solution or if it is better to go with @Paul 's solution, but here I try to fix your regex:

serialNumber = re.search("Serial Number</td><td>(.*?)</td>", source, re.S | re.I )

I removed the \n, because it is difficult in my opinion (\n,\r,\r\n, ...?), instead I used the option re.S (Dotall).

But be aware, now if there is a newline, it will be in your capturing group! i.e. you should strip whitespaces afterwards from your result.

Another problem of your regex is the <TD> in your string but you search for <td>. There for is the option re.I (IgnoreCase).

You can find more explanations about regex here on docs.python.org

Retrieving string from html on non-unique table

2 Answers2