HTMLUnit : different code from when view source in Chrome

Question

I'm using HTMLUnit for get a page on a web. In this webpage, there is a form. When i'm loading from Chrome and view source : something like this :

<form name="form" method="post" onsubmit="return checkDate();">
     <input name="check_in_date" id="check_in_date" readonly="readonly" type="text" class="hasDatepicker"/>
     <input name="check_out_date" id="check_out_date" readonly="readonly" type="text" class="hasDatepicker"/>
     <input name="check_availability" value="test condition" type="submit"/>
</form>

But when I'm loading by using HTMLUnit by this code:

String url = "sample link";
WebClient webClient = new WebClient();
webClient.getOptions().setJavaScriptEnabled(true);
webClient.getOptions().setCssEnabled(false);
webClient.setAjaxController(new NicelyResynchronizingAjaxController());
HtmlPage page = webClient.getPage(url);
System.out.println(page.asXML());

I get different HTML Code. More detail :

<form name="form" method="post" onsubmit="return checkDate();">
         <input name="check_in_date" id="check_in_date" readonly="readonly" type="text" class="hasDatepicker"/>
         <input name="check_out_date" id="check_out_date" readonly="readonly" type="text" class="hasDatepicker"/>
         <input name="check_availability" value="test condition" type="text"/>
 </form>

The different here is : last line : <input name="check_availability" value="test condition" type="text"/> Type now text, instead of submit, so I cannot this kind of code:

HtmlForm form = page.getFormByName("form");
HtmlSubmitInput submit = form.getInputByName("check_availability"); // error at this line
page = submit.click();

Error because now, this field is not a button anymore, it's just a text`. I don't know why there's this difference. Please tell me how to fix this.

Thanks :)

score 1 · Accepted Answer · answered Sep 18 '13 at 20:23

There are various ways to submit a html form using htmlunit.

Create a submit button and append to the form and then click on it. Something like this :

HtmlElement ele=page.createElement("input");
ele.setAttribute("type","submit");
form.appendChild(ele);
ele.click();

References : First - Html Page

Second DomNode

Third - DomElement

2nd Method

Just fire javascript to submit the form

String javaScriptCode="document.getElementById("formid").submit()";
Object result = page.executeJavaScript(javaScriptCode).getJavaScriptResult();

The 2nd method won't work, since there's no ID associated with the form, but for completeness of the answer I've mentioned it.

Caution : The above codes are not tested, but shall work fine.

score 1 · Answer 2 · edited May 23 '17 at 11:50

A few comments before answering your question:

It makes sense that the content you see in Chrome inspector is different from the code returned by HtmlUnit. Furthermore, the code that you see in the Chrome inspector will be different from the code you see when seeing the source code of the page itself (CTRL+U), as the former has already been executed Javascript.

HtmlUnit's javascript engine is different from Chrome's so even if you have enabled JavaScript in HtmlUnit you might get different results. Even the asXml() performs formatting changes in the code.

Having said that I can think of 2 options:

You're comparing a page that has been processed javascript against one that hasn't
You've found a bug

I bet it is the first one. To help you debug play with enabling/disabling JavaScript as well as comparing the result of asXml() with the result of the actual content of the page that HtmlUnit is downloading (I mean, without an processing). To find out more about how to do that check out this question:

How to get the pure raw HTML of a page in HTMLUnit while ignoring JavaScript and CSS?

If you effectively find out that it is HtmlUnit's fault and that the processing of the webpage is failing, please, file a new bug in the SourceForge project. I'll also would like you to provide the test case to reproduce this.

HTMLUnit : different code from when view source in Chrome

2 Answers2