63

I want to use the HTML agility pack to parse tables from complex web pages, but I am somehow lost in the object model.

I looked at the link example, but did not find any table data this way. Can I use XPath to get the tables? I am basically lost after having loaded the data as to how to get the tables. I have done this in Perl before and it was a bit clumsy, but worked. (HTML::TableParser).

I am also happy if one can just shed a light on the right object order for the parsing.

Nathaniel Ford
  • 20,545
  • 20
  • 91
  • 102
weismat
  • 7,195
  • 3
  • 43
  • 58

5 Answers5

128

How about something like: Using HTML Agility Pack

HtmlDocument doc = new HtmlDocument();
doc.LoadHtml(@"<html><body><p><table id=""foo""><tr><th>hello</th></tr><tr><td>world</td></tr></table></body></html>");
foreach (HtmlNode table in doc.DocumentNode.SelectNodes("//table")) {
    Console.WriteLine("Found: " + table.Id);
    foreach (HtmlNode row in table.SelectNodes("tr")) {
        Console.WriteLine("row");
        foreach (HtmlNode cell in row.SelectNodes("th|td")) {
            Console.WriteLine("cell: " + cell.InnerText);
        }
    }
}

Note that you can make it prettier with LINQ-to-Objects if you want:

var query = from table in doc.DocumentNode.SelectNodes("//table").Cast<HtmlNode>()
            from row in table.SelectNodes("tr").Cast<HtmlNode>()
            from cell in row.SelectNodes("th|td").Cast<HtmlNode>()
            select new {Table = table.Id, CellText = cell.InnerText};

foreach(var cell in query) {
    Console.WriteLine("{0}: {1}", cell.Table, cell.CellText);
}
peinearydevelopment
  • 11,042
  • 5
  • 48
  • 76
Marc Gravell
  • 1,026,079
  • 266
  • 2,566
  • 2,900
  • 1
    Hi Mark, can you please advice on parsing large files? Over 50 mb files, I can't fetch child tr nodes on larger files. – Johnny_D Jul 16 '12 at 10:09
  • @Marc - if the table is paging, then how to go to the next page by scraping? – Dark_Knight Jan 12 '17 at 19:12
  • @Dark_Knight you'd need to hit whatever ajax routes the original page used – Marc Gravell Jan 13 '17 at 09:43
  • @MarcGravell i found this `paging_init('sites', 'sites_tbl','/ipID/23.227.38.0/ipIDii/23.227.38.255/sort/6/asc/1', true, '1', '536', {sortCol: '6', sortAsc: '1'})` do you know how to call that function? – Dark_Knight Jan 19 '17 at 09:53
30

The most simple what I've found to get the XPath for a particular Element is to install FireBug extension for Firefox go to the site/webpage press F12 to bring up firebug; right select and right click the element on the page that you want to query and select "Inspect Element" Firebug will select the element in its IDE then right click the Element in Firebug and choose "Copy XPath" this function will give you the exact XPath Query you need to get the element you want using HTML Agility Library.

Coda
  • 517
  • 5
  • 10
  • 4
    Keep in mind that sometimes the browser will change the DOM of the html slightly - like adding to a if it's missing. Html Agility Pack by default will also not include and
    – Anders Sep 26 '11 at 15:09
  • Even acknowledging the pitfalls mentioned by Anders, this is a great time saver. – Phill Healey Jan 02 '15 at 19:07
  • Seems it's not longer supported by Firefox though :( – Noctis Sep 10 '17 at 10:17
3

I know this is a pretty old question but this was my solution that helped with visualizing the table so you can create a class structure. This is also using the HTML Agility Pack

HtmlDocument doc = new HtmlDocument();
doc.LoadHtml(@"<html><body><p><table id=""foo""><tr><th>hello</th></tr><tr><td>world</td></tr></table></body></html>");
var table = doc.DocumentNode.SelectSingleNode("//table");
var tableRows = table.SelectNodes("tr");
var columns = tableRows[0].SelectNodes("th/text()");
for (int i = 1; i < tableRows.Count; i++)
{
    for (int e = 0; e < columns.Count; e++)
    {
        var value = tableRows[i].SelectSingleNode($"td[{e + 1}]");
        Console.Write(columns[e].InnerText + ":" + value.InnerText);
    }
Console.WriteLine();
}
B. Miller
  • 66
  • 3
1

In my case, there is a single table which happens to be a device list from a router. If you wish to read the table using TR/TH/TD (row, header, data) instead of a matrix as mentioned above, you can do something like the following:

    List<TableRow> deviceTable = (from table in document.DocumentNode.SelectNodes(XPathQueries.SELECT_TABLE)
                                       from row in table?.SelectNodes(HtmlBody.TR)
                                       let rows = row.SelectSingleNode(HtmlBody.TR)
                                       where row.FirstChild.OriginalName != null && row.FirstChild.OriginalName.Equals(HtmlBody.T_HEADER)
                                       select new TableRow
                                       {
                                           Header = row.SelectSingleNode(HtmlBody.T_HEADER)?.InnerText,
                                           Data = row.SelectSingleNode(HtmlBody.T_DATA)?.InnerText}).ToList();
                                       }  

TableRow is just a simple object with Header and Data as properties. The approach takes care of null-ness and this case:

<tr>
    <td width="28%">&nbsp;</td>
</tr>

which is row without a header. The HtmlBody object with the constants hanging off of it are probably readily deduced but I apologize for it even still. I came from the world where if you have " in your code, it should either be constant or localizable.

-1

Line from above answer:

HtmlDocument doc = new HtmlDocument();

This doesn't work in VS 2015 C#. You cannot construct an HtmlDocument any more.

Another MS "feature" that makes things more difficult to use. Try HtmlAgilityPack.HtmlWeb and check out this link for some sample code.

Nathaniel Ford
  • 20,545
  • 20
  • 91
  • 102
rk42
  • 7
  • 1