1

I am trying to load the webpage, http://www.artstation.com/artist/nicotine, so I can scrub the page, unfortunately the page seems to be generated via code so the tags that I am looking for aren't available.

Loading it with the following isn't working, as it only loads the source javascript, not the content it generates:

HtmlWeb htmlWeb = new HtmlWeb();
imagepage = htmlWeb.Load(http://www.artstation.com/artist/nicotine);

How can I load the page the page that is shown in the browser so that I can scrub it for the tags?

Chris L
  • 103
  • 1
  • 12
  • I'm not sure what scrubbing is. If you're just interested in getting the HTML, load the webpage in a browser and use `document.body.innerHTML`. You can [copy the content to clipboard](http://stackoverflow.com/questions/400212/how-to-copy-to-the-clipboard-in-javascript) as well. – sampathsris Aug 17 '14 at 05:24
  • You should use [Fiddler](http://www.telerik.com/fiddler) to watch the traffic that goes by as the page loads. Beyond that, read the JavaScript in your code. – Matthew Haugen Aug 17 '14 at 05:32
  • Admittedly, I do think it's an ethical responsibility to point out, that site doesn't look like it welcomes scrubbing like you're doing. It also seems like, by doing it dynamically, they're taking particular steps *in order* to make it hard to do. – Matthew Haugen Aug 17 '14 at 05:33

1 Answers1

0

You can not use HtmlAgilityPack for this. When HAP is asking the server to give you the page file, this file's content hasn't yet been parsed/executed by a web browser and so the JavaScript on it hasn't yet done anything.

There is a work around for this. You can use selenium or phantomJs to get the content of dynamically generated tags. These tools have browser stack and it will execute you the JavaScript. You can find many other tools like this and plenty of examples.

cdev
  • 5,043
  • 2
  • 33
  • 32