C# web request - how to wait until page and all of its subsequent jquery/ajax calls have completed and page has fully loaded

Question

Quick question, hopefully simple, I am using an HttpClient to make a request out to a website and parsing the data on the page, but a lot of the content on the page are modules that reach out after going to the page to grab their data, so when I try to parse it the data is not populated yet and/or is blank. Is there a way to completely wait until the page and all of its page contents on the page load fully before proceeding? If so can you place provide an example.

If there is a better way to accomplish this than using the HttpClient I would be willing to change it to accomplish the task, please let me know.

Thank you in advance for your help.

Environment: Visual Studio 2017, C# .Net Core

are the modules loaded via ajax? – Andrew Reyes Apr 29 '17 at 19:19 — Andrew Reyes, Apr 29 '17 at 19:19

score 2 · Accepted Answer · edited May 23 '17 at 10:31

2

HttpClient will only request the specific URL you asked for. Unlike a browser, it will not parse that page and then follow the links to CSS, script and image files etc, nor will it execute any JS that is on the original page.

You could look into doing this with Selenium and PhantomJS, as proposed in the following answer: https://stackoverflow.com/a/24289395/1134217

edited May 23 '17 at 10:31

Community

1
1

answered Apr 29 '17 at 19:25

Daniel Hume

437
4
9

C# web request - how to wait until page and all of its subsequent jquery/ajax calls have completed and page has fully loaded

1 Answers1