Extracting td/tr from html in bash?

Question

I got the page http://www.cpubenchmark.net/cpu_list.php and I want to extract given CPU's with their Name, Rank and benchmark score.

Example ("Intel Core i5"):

Intel Core i5-3450 @ 3.10GHz - Score: 3333 - Rank: 1
Intel Core i5-3450S @ 2.80GHz - Score: 2222 - Rank: 2
Intel Core i5-2380P @ 3.10GHz - Score: 1111 - Rank: 3
...

How can I do that in bash? Tried to start with something like that (without cpu filtering - don't know how that works):

#!/bin/sh
curl http://www.cpubenchmark.net/cpu_list.php | grep '^<TR><TD>' \
| sed \
    -e 's:<TR>::g'  \
    -e 's:</TR>::g' \
    -e 's:</TD>::g' \
    -e 's:<TD>: :g' \
| cut -c2- >> /home/test.txt

Output is something like that:

<A HREF="cpu_lookup.php?cpu=686+Gen&amp;id=1495">686 Gen</A> 288 1559 NA NA
<A HREF="cpu_lookup.php?cpu=AMD+A10-4600M+APU&amp;id=10">AMD A10-4600M APU</A> 3175 388 NA NA
<A HREF="cpu_lookup.php?cpu=AMD+A10-4655M+APU&amp;id=11">AMD A10-4655M APU</A> 3017 406 NA NA

`Bash` definitely ain't a good tool to parse `HTML`. You sure you can't use something else? (`python` or `perl` advised by i put on my robe an wizard hat) — davak, Dec 28 '12 at 23:54
I mean you can manually sed/cut/tr/... through that but this would basically mean reinventing the wheel... — davak, Dec 28 '12 at 23:56
@BeniBela see "Example ("Intel Core i5")" <- That's how it should look. — recon, Dec 28 '12 at 23:56
@davak python/perl is not installed - so no. Well php is available. — recon, Dec 28 '12 at 23:57
If php is an option for you, you might wanna look here: http://stackoverflow.com/questions/3627489/php-parse-html-code — davak, Dec 28 '12 at 23:59
Looks like another instance where this is a great answer: http://stackoverflow.com/a/1732454/1106534 — Paul Armstrong, Dec 29 '12 at 00:03

score 4 · Accepted Answer · answered Dec 29 '12 at 00:13

If you want to download an additional program you can use my Xidel:

All CPUs:

xidel http://www.cpubenchmark.net/cpu_list.php -e '//table[@id="cputable"]//tr/concat(td[1], " - Score: ", td[2], " - Rank: ", td[3])'

Those starting with Intel...:

xidel http://www.cpubenchmark.net/cpu_list.php -e '//table[@id="cputable"]//tr[starts-with(td[1], "Intel Core i5")]/concat(td[1], " - Score: ", td[2], " - Rank: ", td[3])'

It can even sort them for rank (never used that feature before):

xidel http://www.cpubenchmark.net/cpu_list.php -e 'for $row in //table[@id="cputable"]//tr[starts-with(td[1], "Intel Core i5")] order by $row/td[3] return $row/concat(td[1], " - Score: ", td[2], " - Rank: ", td[3])' --extract-kind=xquery

That's amazing. Never heard about Xidel. Thanks for that nice solution. Works as expected. — recon, Dec 29 '12 at 00:29
Well, I had it sitting as fifth-complete library on my harddisk for almost four years, before I told anyone about it... — BeniBela, Dec 29 '12 at 00:33

score 0 · Answer 2 · answered Dec 29 '12 at 00:38

A bash solution tailored strictly to the current format of the page:

#! /bin/bash

function nextcell
{
    cell=${line%%</TD>*}
    # remove closing link tag if any
    cell=${cell%</?>}
    cell=${cell##*>}
    line=${line#*</TD>}
}

while read line
do
    if [[ ! "$line" =~ cpu_lookup.php ]]
    then
        continue
    fi
    nextcell
    echo -n "$cell"
    nextcell
    echo -n " - Score: $cell"
    nextcell
    echo " - Rank: $cell"
done

Extracting td/tr from html in bash?

2 Answers2