1

I want to make a site which would work with some data from another site. The following code allows me to get the whole DOM from the target.

public IActionResult About(){
  string s = new System.Net.WebClient().DownloadString("https://siteToRead.com");
  ViewData["pageinfo"] = s;
  return View();
}

The problem is that it does not wait until the page is fully loaded (which in my case takes about a second) and therefore there is information that is simply not there yet. In this case it is replaced with "Loading data from server."

Is there a way to give the webClient some time to let the page content load completely?

Shuumi
  • 173
  • 9
  • 3
    I'm pretty sure that site is loading its content dynamically with several ajax calls, triggered by javascript. None of that is handled by the webclient class. Look into a webscraper component. [HtmlAgilityPack](https://stackoverflow.com/questions/846994/how-to-use-html-agility-pack) seems to be popular but other headless browser components exist as well – rene Oct 04 '18 at 10:31
  • for asp.net: https://stackoverflow.com/q/13240915/6132978 – hrnjan Oct 04 '18 at 12:00
  • @hrnjan the question isn't how to make calls asynchronously. – Panagiotis Kanavos Oct 04 '18 at 12:04
  • WebClient only loads the html, if it's manipulated by JavaScript, it won't Work. – Poul Bak Oct 04 '18 at 12:24
  • 1
    You could use the WebBrowser class to *navigate* to that page. You'll probably need a couple of *hacks* to make it work as expected. See this answer: [How can I get the WebBrowser control to show modern contents?](https://stackoverflow.com/questions/38514184/how-can-i-get-the-webbrowser-control-to-show-modern-contents?answertab=active#tab-top). Quite simple to apply. Since it just requires a Registry update in the User section, you should be able to set it up without troubles. (If you open that Registry key, you'll most probably find other softwares listed there. Acrobat Reader, for example). – Jimi Oct 04 '18 at 12:58
  • @rene Thank you for that answer. I was working on it for a few hours and tried 3 different webscrapers. I don't seem able to do it even with a Webbrowser simulator from ScrapySharp. I sent the browser onto a page, left it sitting for 10 seconds and then read the content and it still was "Loading data from server." Now i know what I whould be working with to solve the problem. Thank you very much – Shuumi Oct 04 '18 at 13:42

0 Answers0