HtmlUnit form refresh with Ajax does not work

Question

I try to fill and submit an HTML form using HtmlUnit. One select element and its options are loaded using <body onLoad="...">.

My problem: I cannot retrieve this select element via getSelectByName, getChildElements etc. (ElementNotFoundException is thrown), although I can see that the data has been loaded when looking at the org.apache.http.wire log.

When printing page.asXml(), I see only the unaltered HTML document.

My code:

public static void main(final String[] args) throws Exception {

    final URL url = new URL("http://www.rce-event.de/modules/meldung/annahme.php?oid=471&pid=1&ac=d98482bbf174f62eaaa4664c&tkey=468&portal=www.dachau.de&ortsbox=1&callpopup=1");

    final WebClient webClient = new WebClient(BrowserVersion.FIREFOX_3_6); // tried also FIREFOX_3
    webClient.setAjaxController(new NicelyResynchronizingAjaxController());

    final HtmlPage page = webClient.getPage(url);
    webClient.waitForBackgroundJavaScript(10000); // tried also Thread.sleep()

    // tried also to use webClient.getCurrentWindow().getEnclosedPage() instead of 'page'
    final HtmlForm form = page.getFormByName("formular");

    // ElementNotFoundException thrown here:
    final HtmlSelect select = form.getSelectByName("event.theme");
    final HtmlOption option = select.getOptionByText("Sport/Freizeit");
    final Page newPage = select.setSelectedAttribute(option, false);

    // submit etc.
}

Stacktrace:

Exception in thread "main" com.gargoylesoftware.htmlunit.ElementNotFoundException: elementName=[select] attributeName=[name] attributeValue=[event.theme]
at com.gargoylesoftware.htmlunit.html.HtmlForm.getSelectByName(HtmlForm.java:449)
at Xyzzy.main(Xyzzy.java:58)

I tried everything written here, here, and here (and even more), without any success.

Update:

I simplified my code and started a bounty.

can you provide whole code with line number or specify in what number exception generate? — Dmitry, May 21 '12 at 21:01
The execption occurs on this call: `getSelectByName("event.theme")` (it's already commented) — michael667, May 21 '12 at 21:07
are you shure you have such lement on you page? possibly it is not visible/inside frame/iframe and that is the case? — Dmitry, May 21 '12 at 21:15
Just look at the source code in a real browser (using Firebug or something similar) — michael667, May 21 '12 at 21:17
let us [continue this discussion in chat](http://chat.stackoverflow.com/rooms/11534/discussion-between-michael667-and-dmitry) — michael667, May 21 '12 at 21:33
I would suspect that perhaps htmlUnit isn't up to the complex asynchronous way that you create your select. It does say Async http requests, but onreadystatechange ?? not sure. Perhaps for debugging, try to get the `kat_content` element, and check (1) the innerHTML and (2) the child elements (enumerate them) and see what you get back. Alternatively, how about creating the select initially, and only populating it when you get the data back. You may have more luck that way. — GregHNZ, May 24 '12 at 09:43
kat_content is empty, just like in the initial HTML page. It seems that the page DOM is never updated within HtmlUnit. — michael667, May 24 '12 at 09:46

score 2 · Answer 1 · answered May 30 '12 at 20:36

Your problem is that the select named "event.theme" only gets loaded once the select named "event.datapool" has a value of "1" as selected.

So you need to change the "event.datapool" select value to "1" :

[........]
final HtmlSelect selectBase = form.getSelectByName("event.datapool");
final HtmlOption optionBase = selectBase.getOptionByText("Freizeit / Tourismus");
final Page newPage = selectBase.setSelectedAttribute(optionBase, true);
[........]

But you may have problems because the "HTML" data for the select "event.theme" is loaded via ajax. So I do not think your java "HtmlSelect" class will loaded the select "event.theme" in the form as Javascript does with an actual user interation.

A solution to that would be to:

1. Load your page "http://www.rce-event.de/modules/meldung/annahme.php?oid=471&pid=1&ac=d98482bbf174f62eaaa4664c&tkey=468&portal=www.dachau.de&ortsbox=1&callpopup=1" 
2. Load the page "http://www.rce-event.de/modules/meldung/js/xmlhttp_querys.php?get_kat=1&time=1338409551228&id=1&block=kat" > which will return the "event.theme" select data/values
3. Then use the data loaded in step 2 to update the page loaded in step 1 by inserting a "select list with id and name set to <event.theme>" in the HTML element "kat_content"

Then your form/loaded webpage should have the new select named "event.theme" and therfore the following code shouldn't produce errors anymore.

final HtmlSelect select = form.getSelectByName("event.theme");
final HtmlOption option = select.getOptionByText("Sport/Freizeit");
final Page newPage = select.setSelectedAttribute(option, false);

Your solution means I should implement in Java what the `Datensubkatloc` Javascript function included in the page is already doing. Why isn't that method executed properly by HtmlUnit in the first place, just as it is executed by any other browser? — michael667, May 31 '12 at 06:51

HtmlUnit form refresh with Ajax does not work

1 Answers1