8

Trying to parse an HTML document and extract some elements (any links to text files).

The current strategy is to load an HTML document into a string. Then find all instances of links to text files. It could be any file type, but for this question, it's a text file.

The end goal is to have an IEnumerable list of string objects. That part is easy, but parsing the data is the question.

<html>
<head><title>Blah</title>
</head>
<body>
<br/>
<div>Here is your first text file: <a href="http://myServer.com/blah.txt"></div>
<span>Here is your second text file: <a href="http://myServer.com/blarg2.txt"></span>
<div>Here is your third text file: <a href="http://myServer.com/bat.txt"></div>
<div>Here is your fourth text file: <a href="http://myServer.com/somefile.txt"></div>
<div>Thanks for visiting!</div>
</body>
</html>

The initial approaches are:

  • load the string into an XML document, and attack it in a Linq-To-Xml fashion.
  • create a regex, to look for a string starting with href=, and ending with .txt

The question being:

  • what would that regex look like? I am a regex newbie, and this is part of my regex learning.
  • which method would you use to extract a list of tags?
  • which would be the most performant way?
  • which method would be the most readable/maintainable?

Update: Kudos to Matthew on the HTML Agility Pack suggestion. It worked just fine! The XPath suggestion works as well. I wish I could mark both answers as 'The Answer', but I obviously cannot. They are both valid solutions to the problem.

Here's a C# console app using the regex suggested by Jeff. It reads the string fine, and will not include any href that is not ended with .txt. With the given sample, it correctly does NOT include the .txt.snarg file in the results (as provided in the HTML string function).

using System;
using System.Collections.Generic;
using System.Text;
using System.Text.RegularExpressions;
using System.IO;

namespace ParsePageLinks
{
    class Program
    {
        static void Main(string[] args)
        {
            GetAllLinksFromStringByRegex();
        }

        static List<string> GetAllLinksFromStringByRegex()
        {
            string myHtmlString = BuildHtmlString();
            string txtFileExp = "href=\"([^\\\"]*\\.txt)\"";

            List<string> foundTextFiles = new List<string>();

            MatchCollection textFileLinkMatches = Regex.Matches(myHtmlString, txtFileExp, RegexOptions.IgnoreCase);
            foreach (Match m in textFileLinkMatches)
            {
                foundTextFiles.Add( m.Groups[1].ToString()); // this is your captured group
            }

            return files;
        }

            static string BuildHtmlString()
            {
                return new StringReader(@"<html><head><title>Blah</title></head><body><br/>
<div>Here is your first text file: <a href=""http://myServer.com/blah.txt""></div>
<span>Here is your second text file: <a href=""http://myServer.com/blarg2.txt""></span>
<div>Here is your third text file: <a href=""http://myServer.com/bat.txt.snarg""></div>
<div>Here is your fourth text file: <a href=""http://myServer.com/somefile.txt""></div>
<div>Thanks for visiting!</div></body></html>").ReadToEnd();
            }       
        }
    }
Community
  • 1
  • 1
p.campbell
  • 98,673
  • 67
  • 256
  • 322

4 Answers4

13

Neither. Load it into an (X/HT)MLDocument and use XPath, which is a standard method of manipulating XML and very powerful. The functions to look at are SelectNodes and SelectSingleNode.

Since you are apparently using HTML (not XHTML), you should use HTML Agility Pack. Most of the methods and properties match the related XML classes.

Sample implementation using XPath:

    HtmlDocument doc = new HtmlDocument();
    doc.Load(new StringReader(@"<html>
<head><title>Blah</title>
</head>
<body>
<br/>
<div>Here is your first text file: <a href=""http://myServer.com/blah.txt""></div>
<span>Here is your second text file: <a href=""http://myServer.com/blarg2.txt""></span>
<div>Here is your third text file: <a href=""http://myServer.com/bat.txt""></div>
<div>Here is your fourth text file: <a href=""http://myServer.com/somefile.txt""></div>
<div>Thanks for visiting!</div>
</body>
</html>"));
        HtmlNode root = doc.DocumentNode;
        // 3 = ".txt".Length - 1.  See http://stackoverflow.com/questions/402211/how-to-use-xpath-function-in-a-xpathexpression-instance-programatically
        HtmlNodeCollection links = root.SelectNodes("//a[@href['.txt' = substring(., string-length(.)- 3)]]");
    IList<string> fileStrings;
    if(links != null)
    {
        fileStrings = new List<string>(links.Count);
        foreach(HtmlNode link in links)
        fileStrings.Add(link.GetAttributeValue("href", null));
    }
    else
        fileStrings = new List<string>(0);
Matthew Flaschen
  • 278,309
  • 50
  • 514
  • 539
1

I would recommend regex. Why?

  • Flexible (case-insensitivity, easy to add new file extensions, elements to check, etc.)
  • Fast to write
  • Fast to run

Regex expressions will not be hard to read, as long as you can WRITE regexes.

using this as the regular expression:

href="([^"]*\.txt)"

Explanation:

  • It has parentheses around the filename, which will result in a "captured group" which you can access after each match has been found.
  • It has to escape the "." by using the regex escape character, a backslash.
  • It has to match any character EXCEPT double-quotes: [^"] until it finds
    the ".txt"

it translates into an escaped string like this:

string txtExp = "href=\"([^\\\"]*\\.txt)\"

Then you can iterate over your Matches:

Matches txtMatches = Regex.Matches(input, exp, RegexOptions.IgnoreCase);
foreach(Match m in txtMatches) {
  string filename = m.Groups[1]; // this is your captured group
}
Jeff Meatball Yang
  • 37,839
  • 27
  • 91
  • 125
  • 1
    @Jeff: this is an excellent code sample. Thank you for the input! – p.campbell May 25 '09 at 18:40
  • 4
    That will match .txt anywhere in the href, when the OP explicitly said "ends with". In my opinion, regex is inappropriate here. – Matthew Flaschen May 25 '09 at 19:01
  • @Matthew: No, It will only match an HREF ending with (.txt"). I don't think HREF's contain quotes in the middle. – Dmitri Farkov May 25 '09 at 19:47
  • 1
    Don't try to use regular expressions to parse non-regular languages. – Svante May 25 '09 at 20:42
  • 1
    I understand the desire to approach this from a DOM/XPath point of view - but my rationale was that a regex implementation assumes very little about the input data. Obviously, if the OP can make assumptions, especially like well-formed documents, a DOM approach is much "cleaner". @Svante: I think regexes are GREAT at finding known patterns out of non-regular data. Think how many times you've grepped for something with a regex. Also, the OP wanted a regex example. – Jeff Meatball Yang May 26 '09 at 16:05
  • I misread your regex. However, href's actually can contain "'s, if they are surrounded by '. foo Granted that is somewhat perverse, but it does validate. Anyway, the important thing is that the OP got something that works well for their current data. – Matthew Flaschen May 26 '09 at 21:18
0

REGEX is not fast, in fact it's slower than native string parse stuff in .NET. Don't believe me, see for yourself.

None of the examples above are faster than going to the DOM directly. 

HTMLDocument doc = wb.Document;
var links = doc.Links;
RichardTheKiwi
  • 105,798
  • 26
  • 196
  • 262
JWP
  • 25
  • 1
0

Alternatively to Matthew Flaschen's suggestion, DOM (e.g. if you suffer from a X?L allergy outbreak)

It gets a bad rep sometimes - I guess because implementations are funny sometimes, and the native COM interfaces are a bit unwieldy without some (minor) smart helpers, but I've found it a robust, stable and intuitive / explorable way to parse and manipulate HTML.

peterchen
  • 40,917
  • 20
  • 104
  • 186