0

I know there is another question with practically identical title here: Webclient.DownloadString does not retrieve the whole page But the solution doesn't help me, maybe somebody else have the same problem.

I'm trying to get the html code of this URL:

https://cubebrush.co/?freebies=true

To achieve that, I'm using the following code in C#:

WebClient webClient = new WebClient();
string webString = webClient.DownloadString("https://cubebrush.co/?freebies=true");

But the retrieved html lacks some information, for example, all the button tags inside the website. This can be quickly checked using the library HtmlAgilityPack and checking for all the tags inside the website with the following code:

HtmlDocument doc = new HtmlDocument();
doc.LoadHtml(webString);
HashSet<string> hs = new HashSet<string>();
foreach (var dec in doc.DocumentNode.Descendants())
{
    hs.Add(dec.Name);
}

If we run this, it will show 26 tags, but none of them will be a button tag. This makes sense, since the initial webString also lacks that "button information".

I've tried to copy webString into a file, to check if, as the initial commented post says, was a problem with the visualizer, but it doesn't, visualizer and file looks exactly equal.

Can somebody tells me what I'm doing wrong? Thanks!

Lotan
  • 4,078
  • 1
  • 12
  • 30
  • Its worth a note, WebClient / HttpClient doesnt render the page and all its scripts. Also the server might give you a different page based on your user agent – TheGeneral Aug 07 '20 at 04:07
  • @TheGeneral thanks for the comment, but could you expalain it a little further please? Are you telling me that what HttpClient retrieve, is only the "initially rendered website", that could not have all the elements? But then how can I wait until It render all elements? – Lotan Aug 07 '20 at 07:51
  • It depends. Depending on the site, some of the content may be dynamically created via the scripts on the page (this may or may not be the case). If it is the case, you will have to use a library that can render the page like a browser. Ie like selenium ect. – TheGeneral Aug 07 '20 at 08:03

0 Answers0