c# web scraping to get URL from html

Question

I am trying to scrape a website and get a URL from it, I am using htmlagilitypack and the code below:

HtmlWeb hw = new HtmlWeb();
HtmlAgilityPack.HtmlDocument doc = hw.Load("http://putlocker.ist/scorpion-season-1-episode-1/");

foreach (HtmlNode link in doc.DocumentNode.SelectNodes("//[@id='vplayer_media']/video"))
{
    string whatever = link.OuterHtml;
    textBox1.Text = whatever;
}

I used opera's developer tools to copy the XPath which came out as this:

//*[@id="vplayer_media"]/video

I couldn't use it because of double quotes so I replaced it with

@"//*[@id=""vplayer_media""]/video"

but I get the error:

Object reference not set to an instance of an object

What am I doing wrong?

If you copy/paste stuff, make sure you copy everything. You missed the * — rene, Jul 02 '17 at 09:14
I guess, single quote and double quote work the same for XPath, but I hope you understand that you can insert a double quote in a string by escaping it like this `\"`. — Yeldar Kurmangaliyev, Jul 02 '17 at 09:15
oops, thanks @rene, fixed it, but now I get object reference not set to an instance of an object — Escage NULL, Jul 02 '17 at 09:18
https://stackoverflow.com/questions/4660142/what-is-a-nullreferenceexception-and-how-do-i-fix-it — rene, Jul 02 '17 at 09:20
I give it a parameter though, I must be doing it incorrectly — Escage NULL, Jul 02 '17 at 09:22
thats odd @AlexanderPetrov I am not logged in, I tried that class above it: @"//*[@id=""vplayer_media""]" and still got the same error — Escage NULL, Jul 02 '17 at 09:39
There is no `id='vplayer_media'` attribute on the page without login. You must be logged in to get access to the real data. — Alexander Petrov, Jul 02 '17 at 09:49

score 0 · Accepted Answer · answered Jul 02 '17 at 09:14

0

Escape the double-quotes in your xpath:

"//[@id=\"vplayer_media\"]/video"

Or use double-double-quotes with a 'literal'

@"//[@id=""vplayer_media""]/video"

answered Jul 02 '17 at 09:14

PhonicUK

13,486
4
43
62

I get object reference not set to an instance of an object – Escage NULL Jul 02 '17 at 09:15
the problem is that the xpath is not valid. There is a * missing and with that the question is a simply typo problem. – rene Jul 02 '17 at 09:21
(@"//*[@id=""vplayer_media""]/video") I added it in, how is it still incorrect? – Escage NULL Jul 02 '17 at 09:25
Single quote `'` completely legal in xpath. – Alexander Petrov Jul 02 '17 at 09:35

c# web scraping to get URL from html

1 Answers1