How to extract a url from a webpage?

Question

Twitter returns a webpage that these lines are among others:

<link rel="dns-prefetch" href="//video.twimg.com" />
<link rel="preload" as="script" crossorigin="anonymous" href="https://ma-0.twimg.com/twitter-assets/responsive-web/web/ltr/vendor.69f9ac19fa493004.js" />
<link rel="preload" as="script" crossorigin="anonymous" href="https://ma-0.twimg.com/twitter-assets/responsive-web/web/ltr/i18n/en.312d3f56908013c9.js" />
<link rel="preload" as="script" crossorigin="anonymous" href="https://ma-0.twimg.com/twitter-assets/responsive-web/web/ltr/main.da8c0a0fbf03fdac.js" />
<meta property="fb:app_id" content="2231777543" />

I need the url that contains the main.*.js file. How can I get it?

I tried this:

var mainIndex = content.IndexOf("main.");
var startIndex = content.LastIndexOf("href=\"", mainIndex) + 6;
var endIndex = content.IndexOf(".js", startIndex) + 3;
var url = content.Substring(startIndex, endIndex - startIndex);

but it's a bad unsafe implementation. Thanks.

Never,Never Never use Regex to parse a webpage. Very bad suggestion. It is ok to use regex on substrings extracted from html. You should be using GetElementsByTagName("link") then extracting the attributes href which returns a string. — jdweng, Jun 06 '17 at 14:21
Do not use regex, see the HtmlAgilityPack-related answer below. — Wiktor Stribiżew, Jun 06 '17 at 14:46

score 3 · Accepted Answer · answered Jun 06 '17 at 14:37

3

You can do it with a dedicated HTML parser, such as Html Agility Pack

var text = "<link rel=\"dns-prefetch\" href=\"//vide.... />";
HtmlDocument doc = new HtmlDocument();
doc.LoadHtml(text);

var links = doc
    .DocumentNode
    .SelectNodes("//link")
    .Select(e=>e.Attributes["href"].Value);

links
    // here, you could parse and match the URL robustly
    .Where(href=>href.Contains("main"))
    // try it in LINQPad
    .Dump();

result: https://ma-0.twimg.com/twitter-assets/responsive-web/web/ltr/main.da8c0a0fbf03fdac.js

answered Jun 06 '17 at 14:37

Dmitry Ledentsov

3,620
18
28

Thanks @dmitryLedentsov .. I try it now – Blendester Jun 06 '17 at 14:47
sorry it doesn't have a '.SelectNodes' method.. am I missing a using directive? – Blendester Jun 06 '17 at 14:56
@Blendester yes, you need a `using HtmlAgilityPack;` line. Did you install it via Nuget? – Dmitry Ledentsov Jun 06 '17 at 14:57
yes I did using NuGet .. also added using HtmlAgilityPack.. it's inside a class library – Blendester Jun 06 '17 at 15:00
but says `SelectNodes` doesn't exist. isn't any other way to select nodes? – Blendester Jun 06 '17 at 15:06
version: 1.4.9.5. It's a well-documented C# library: what does the IntelliSense/disassembly show you, what the available methods are? – Dmitry Ledentsov Jun 06 '17 at 15:47
`SelectNodes` is event part of the official samples http://html-agility-pack.net/ . Please check your build configuration – Dmitry Ledentsov Jun 06 '17 at 15:49
Let us [continue this discussion in chat](http://chat.stackoverflow.com/rooms/146065/discussion-between-dmitry-ledentsov-and-blendester). – Dmitry Ledentsov Jun 07 '17 at 12:11

score -3 · Answer 2 · answered Jun 06 '17 at 14:16

it is definitely a good idea to use regex for it. first one regex for the prefix part that should be replaced with empty string and same thing for end.
you will need to escape the chars like < with \< and to use the normal regex syntax to exactly define what is required.

How to extract a url from a webpage?

2 Answers2