2

I am trying to use a headless browser in .Net and I need it to execute JavaScript because the information I need to read with it is loaded via AJAX. After some research I believe my best option is HtmlUnit. I have followed the instructions in Steven Sanderson's blog http://blog.stevensanderson.com/2010/03/30/using-htmlunit-on-net-for-headless-browser-automation/ in the main this works great. However, as hard as I try I cannot get it to wait for the JavaScript to load. Essentially after selecting something from a selectbox an AJAX request fires and populated another selectbox. HtmlUnit never sees the new options.

I have tried combinations of all of the examples in the FAQ http://htmlunit.sourceforge.net/faq.html#AJAXDoesNotWork so I can only assume that it is a problem with the fact it has been converted to run in .Net. I am hoping someone has some experience in this who can shed some light on what is wrong. I feel like I am going the right way with HtmlUnit and it has been recommended on here before Is there a headless browser for .NET (for automated Web-UI-Tests)? and headless internet browser? but I have reached a dead end.

So if anyone has either a better choice for a headles browser that supports JavaScript, and ideas how to get HtmlUnit to wait for the JavaScript or even just some advice on how to debug this further.

Thanks.

Simon

Community
  • 1
  • 1
baynezy
  • 6,493
  • 10
  • 48
  • 73
  • I don't suppose you could integrate jQuery and its [document.ready()](http://api.jquery.com/ready/) function? Might take some tweaking but this way the browser would wait for the document to load completely before HtmlUnit tries to do its thing. – Darth Continent Mar 27 '12 at 14:14

1 Answers1

3

I think you need to use webClient.waitForBackgroundJavaScript() or webClient.waitForBackgroundJavaScriptStartingBefore().

You can also see these links that might help you little.

  1. HTMLUnit doesn't wait for Javascript
  2. HtmlUnit doesn't load externally referenced javascript
  3. htmlunit javascript question
Community
  • 1
  • 1
Somnath Muluk
  • 55,015
  • 38
  • 216
  • 226
  • Sadly I have tried both of those and neither work. Because of the evidence pointing to the fact that they should work I can only come the the conclusion that it is something to do with how the website is built and not something I can resolve within HTMLUnit. – baynezy Mar 29 '12 at 06:36