Parse data or data in table through Bash

Question

I would like to parse data from websites through Bash using sed or awk (feel free to change my direction to process data).

Here is a sample of code.

<tbody>
        <tr>
            <td class="text-left">111</td><td class="text-center">
                <a href="/path1.htm">AAA</a>
            </td><td class="text-center">
                <a href="/path2.htm" class="tp-link-policy">BBB</a>
            </td><td class="text-center">
                Updated October, 2016
            </td>
        </tr><tr>
            <td class="text-left">CCC</td><td class="text-center">
                <a href="/path3.htm">
            .
            .
            .
            .
        </tr>
</tbody>

Usually when I used preg_match in PHP I had no problem with newlines, but in Bash I need to change my thinking of regex completely. Do you recommend to first prepare data to be readable by sed and awk to delete all newlines and recreate them again different way based on what structure of data I desire to use sed or awk?

For example, I would create a newline for every <tr> so the result would look like this? Am I right or I should leave this way of thinking? This would work, but I do not feel comfortable to manipulate data like this.

<tbody>
<tr><td class="text-left">111</td><td class="text-center"><a href="/path1.htm">AAA</a></td><td class="text-center"><a href="/path2.htm" class="tp-link-policy">BBB</a></td><td class="text-center">Updated October, 2016</td></tr>
<tr><td class="text-left">CCC</td><td class="text-center"><a href="/path3.htm">....</tr></tbody>

Output should be, for example:

111|AAA|BBB|Updated October, 2016

Forget about using line-based text processing tools such as awk and sed if you want to do anything useful with HTML. — Tom Fenech, Nov 25 '16 at 14:32
For command line HTML parsing, you can look into `html2` (from the `xml2` package), or `hxpipe` and related functions (from the `html-xml-utils` package). They both roughly do what you're trying to do. — Benjamin W., Nov 25 '16 at 14:46
I expected to be wrong in this. Perfect, thank you very much for a helpful insight. @Benjamin W. — Pavol Travnik, Nov 25 '16 at 15:01

Pavol Travnik · Accepted Answer · 2017-02-08T08:39:14.327

0

I have used xmllint after all.

xmllint --html --shell <file>

Then I executed this command to retrieve a demanded xpath.

grep <text>

When you find a structure in your html file you can fully search through your file based on xpath.

xmllint --html --xpath <xpath> <file>

However much efficient is to use Python and Beautiful Soup.

edited Feb 08 '17 at 08:39

answered Nov 25 '16 at 16:02

Pavol Travnik

853
3
14
32

Parse data or data in table through Bash

1 Answers1