HtmlUnit does not find the element

Question

I'm trying to get the textbox with u_0_1e as id, from the page wall but HtmlUnit does not find anything. The last line prints null. Here's the code:

java.util.logging.Logger.getLogger("com.gargoylesoftware").setLevel(java.util.logging.Level.OFF);

        WebClient client = new WebClient(BrowserVersion.CHROME);
        JavaScriptEngine engine = new JavaScriptEngine(client);

        client.setJavaScriptEngine(engine);
        HtmlPage home = client.getPage("https://www.facebook.com/login.php");
        HtmlSubmitInput login = (HtmlSubmitInput) home.getElementById("u_0_1");
        HtmlTextInput name = (HtmlTextInput) home.getElementById("email");
        HtmlPasswordInput pass = (HtmlPasswordInput) home.getElementById("pass");

        name.setValueAttribute("myname");
        pass.setValueAttribute("mypass");

        HtmlPage page = login.click();
        HtmlPage wall = client.getPage("https://www.facebook.com/");

        System.out.println(wall.getElementById("u_0_1e"));

Answer 1

I have some comments about your issue.

First of all, you have disabled HtmlUnit's logging. So if you have any JavaScript issue then you are not going to see it. If you are actually getting a JavaScript error then the JavaScript code won't be fully executed. If the element you're trying to fetch was dynamically fetched from the server (probably using AJAX) then the JavaScript errors, if any, might result in that element not being fetched.

If you are webscraping, which is clearly the case, then you don't have any control over the JS so you can only accept it as not working or disable JS and manually processing the AJAX requests.

Of course, you will see the page perfectly working on a real browser but take into consideration that the JavaScript engine HtmlUnit uses is different from the real browsers.

Secondly, the two lines containing the word engine are absolutely unneeded.

Thirdly, as I mentioned in a previous question of yours, this will be more suitable to be handled by means of the Facebook API.

Finally, you might find this other answer useful: JavaScript not being properly executed in HtmlUnit