Parsing Hyperlinks from a webpage

Question

I have written following code to parse hyperlinks from a given page.

    WebClient web = new WebClient();
    string html = web.DownloadString("http://www.msdn.com");
    string[] separators = new string[] { "<a ", ">" };
    List<string> hyperlinks= html.Split(separators, StringSplitOptions.None).Select(s =>
    {
        if (s.Contains("href"))
            return s;
        else
            return null;
    }).ToList();

Although string split still has to be tweaked to return urls perfectly. My question is there some Data Structure, something on the line of XmlReader or so, which could read HTML strings efficiently.

Any suggestion for improving above code would also be helpful.

Thanks for your time.

hi, just wondering: did you find any useful answers to your question? — Thousand, Sep 28 '12 at 11:32

score 2 · Accepted Answer · answered Sep 26 '12 at 22:15

try HtmlAgilityPack

        HtmlWeb hw = new HtmlWeb();
        HtmlDocument doc = hw.Load("http://www.msdn.com");
        foreach (HtmlNode link in doc.DocumentNode.SelectNodes("//a[@href]"))
        {
            Console.WriteLine(link.GetAttributeValue("href", null));         
        }

this will print out every link on your URL.

if you want to store the links in a list:

 var linkList = doc.DocumentNode.SelectNodes("//a[@href]")
               .Select(i => i.GetAttributeValue("href", null)).ToList();

score 1 · Answer 2 · answered Sep 26 '12 at 21:50

1

You should be using a parser. The most widely used one is HtmlAgilityPack. Using that, you can interact with the HTML as a DOM.

answered Sep 26 '12 at 21:50

Kirk Woll

76,112
22
180
195

score 1 · Answer 3 · edited May 23 '17 at 12:04

1

Assuming you're dealing with well formed XHTML, you could simply treat the text as an XML document. The framework is loaded with features to do exactly what you're asking.

http://msdn.microsoft.com/en-us/library/system.xml.xmldocument.aspx

Does .NET framework offer methods to parse an HTML string?

edited May 23 '17 at 12:04

Community

1
1

answered Sep 26 '12 at 22:15

Doug

37
4

When it is not feasible to install new tools, you have shown a good way. Thank you. – Abhijeet Sep 28 '12 at 11:55

score 0 · Answer 4 · answered Sep 26 '12 at 21:52

0

refactored,

        var html = new WebClient().DownloadString("http://www.msdn.com");
        var separators = new[] { "<a ", ">" };
        html.Split(separators, StringSplitOptions.None).Select(s => s.Contains("href") ? s : null).ToList();

answered Sep 26 '12 at 21:52

Rahul Rumalla

333
1
9

Parsing Hyperlinks from a webpage

4 Answers4