0

I can load a url into a WebBrowser control and perform a login (forms based), I see what I need to see. Great, now I want to use XPath to get the data I need.

Can't do that with a WebBrowser (unless you disagree?) so I use The Agility Pack to kick of a new session as per below:

var wc = new WebClient();
wc.Headers[HttpRequestHeader.UserAgent] = "Mozilla/5.0 (Windows NT 6.3; WOW64; rv:46.0) Gecko/20100101 Firefox/46.0";
var doc = new HtmlAgilityPack.HtmlDocument();
doc.Load(wc.OpenRead(url), Encoding.UTF8);

var value = doc.DocumentNode.SelectSingleNode("|//li[@data-section='currentPositionsDetails']//*[@class='description']");

My value is not retrievable because the website doesn't expose it to the public (it wants a logged in session). How can I "pass on" my WebBrowser control session to my WebClient()? Looking into some of the methods of how to POST my login information, it all seems awfully complicated.

Any ideas? - Thanks

2 Answers2

0

You can retrieve the body html string with webBrowser1.Document.Body.OuterHtml and load it with HtmlAgilityPack:

var doc = new HtmlAgilityPack.HtmlDocument();
doc.Load(new StringReader(this.webBrowser1.Document.Body.OuterHtml));
doomgg
  • 337
  • 2
  • 8
  • Thanks, tried this. But only half my XPath queries are returning a value. The XPath returns a result when I test using a tool called HAPXPathFinder. Can you think of any difference between the OuterHtml returned by the webBrowser control and what a WebClient() might load? –  Jun 10 '16 at 02:24
0

OK, posting this as an answer as it seems to be answered/discussed elsewhere. It's not going to be easy for an amateur like me!

How to pass cookies to HtmlAgilityPack or WebClient?

HtmlAgilityPack.HtmlDocument Cookies

Community
  • 1
  • 1