Get all links on html page?

Question

Im working on a little hobby project. I already have written the code to get a url, download the header and return the mime type / content type.

However, the step before this is the one im stuck on - i need to retrieve the contents of all the urls on the page based inside a tag, and in quotes i.e.

...
<link rel='shortcut icon' href="/static/favicon.ico" type="image/x-icon" />
...

Would find the favicon link.

Is there anything helpful in the .net library or is this going to have to be a case for regex?

I'm getting a weird feeling that the Html Agility Pack is the way to go... — jball, Feb 11 '10 at 23:01

score 74 · Accepted Answer · edited Feb 08 '18 at 12:14

74

I'd look at using the Html Agility Pack.

Here's an example straight from their examples page on how to find all the links in a page:

 HtmlWeb hw = new HtmlWeb();
 HtmlDocument doc = hw.Load(/* url */);
 foreach(HtmlNode link in doc.DocumentNode.SelectNodes("//a[@href]"))
 {

 }

edited Feb 08 '18 at 12:14

Let's Enkindle

57
2
17

answered Feb 11 '10 at 22:56

womp

115,835
26
236
269

Just wanted to add that if the site that you are trying to load has some gzip compression, it will throw an exception on `hw.Load` : `"'gzip' is not a supported encoding name. For information on defining a custom encoding, see the documentation for the Encoding.RegisterProvider method`. Found a workaround [here](https://stackoverflow.com/a/36220920/6178243) – Lucas Palma Stabile Jun 20 '17 at 20:16
Be careful. It's a chance to get NullReferenceException here. ```Returns: An HtmlAgilityPack.HtmlNodeCollection containing a collection of nodes matching the HtmlAgilityPack.HtmlNode.XPath query, or null if no node matched the XPath expression``` – inser Jul 15 '20 at 07:52

SLaks · Answer 2 · 2013-02-22T03:39:05.863

49

You need to use the HTML Agility Pack.

For example:

var doc = new HtmlWeb().Load(url);
var linkTags = doc.DocumentNode.Descendants("link");
var linkedPages = doc.DocumentNode.Descendants("a")
                                  .Select(a => a.GetAttributeValue("href", null))
                                  .Where(u => !String.IsNullOrEmpty(u));

edited Feb 22 '13 at 03:39

answered Feb 11 '10 at 22:57

SLaks

868,454
176
1,908
1,964

Thanks for using a comprehensive and self explanatory LINQ query! – Gonzo345 Jan 15 '19 at 08:38

score 18 · Answer 3 · answered Feb 11 '10 at 22:56

18

There isn't anything built into the BCL, but fortunately you can use the HTML Agility Pack to accomplish this task quite simply.

As for your specific problem, please see Easily extracting links from a snippet of html with HtmlAgilityPack:

private List<string> ExtractAllAHrefTags(HtmlDocument htmlSnippet)
{
    List<string> hrefTags = new List<string>();

    foreach (HtmlNode link in htmlSnippet.DocumentNode.SelectNodes("//a[@href]"))
    {
        HtmlAttribute att = link.Attributes["href"];
        hrefTags.Add(att.Value);
    }

    return hrefTags;
}

answered Feb 11 '10 at 22:56

Andrew Hare

344,730
71
640
635

5

I disagree that a LINQ-based approach is simpler. Declarative? Yes. Functional? Absolutely. Simpler? No, both solutions are equal in their simplicity. – Andrew Hare Feb 11 '10 at 23:14
XPath should use less memory – Don Rolling Nov 18 '14 at 18:01
I would suggest that using XPath would be much less memory intensive. – Don Rolling Nov 18 '14 at 18:02

score 10 · Answer 4 · answered Aug 04 '15 at 09:42

10

How about Regex?

<(a|link).*?href=(\"|')(.+?)(\"|').*?>

with flags IgnoreCase and SingleLine

See demo on systemtextregularexpressions.com regex.matches

answered Aug 04 '15 at 09:42

GRUNGER

486
3
14

This is a better solution than resorting to an entire 3rd party library. – Erick Brown Jun 02 '20 at 17:54

Get all links on html page?

4 Answers4

Linked

Related