Selenium-Java-geckodriver : Test Execution halts with JavaScript error as "document.getElementById(...) is null"

Question

I had been encountering this issue for a while now. Browsed through a lot of threads on SO & other forums but still clueless.

While automating a simple flow on a Web Application with Selenium 3.4.0, geckodriver v0.16.1 & Mozilla Firefox 53.0 within Eclipse Neon v2 IDE, I am facing an error intermittently on the console as:

JavaScript error: https://www.url.com/my, line 1715: TypeError: document.getElementById(...) is null

Though using chromedriver v2.29/Google Chrome 58.0 or using Python I don't face any such issue.

Once this error appears, the Test Execution halts and finally shows TimeoutException as follows:

Exception in thread "main" org.openqa.selenium.TimeoutException: Timeout loading page after 300000ms

URL of the website is: https://www.shareinvestor.com/my

The HTML DOM is:

<div id="sic_sitemap">
  <div id="sic_container">
    <div id="sic_header">
      <h1 id="sic_siteTitle">
        <div id="sic_headerMembershipLink">
          <a id="sic_mobileEdition" href="/mobile">
            <div id="sic_loginContainer" class="sic_logIn" style="">
              <div class="sic_login-wrap">
                <div class="sic_logIn-subscribe">
                  <div class="sic_logIn-bg">
                    <a href="/user/login.html">
                  </div>
                </div>
              </div>
            </div>
            <div id="sic_subHeader">
              <div id="sic_mainNav" class="sic_withRightCorner">
                <div id="sic_sideBar" class="sic_expanded { selected_market_suffix: 'MY'}">
                  <div class="sic_superBanner sic_superBannerTop">
                    <div id="sic_content" class="sic_collapsed">
                      <div id="sic_footer" class="si_fixed">
                      </div>

As of now, I have tried out the following options but of no avail:

1. Java Click
2. JavascriptExecutor Click
3. Actions Click

Here is my code:

import org.openqa.selenium.By;
import org.openqa.selenium.JavascriptExecutor;
import org.openqa.selenium.WebDriver;
import org.openqa.selenium.WebElement;
import org.openqa.selenium.firefox.FirefoxDriver;
import org.openqa.selenium.interactions.Actions;
import org.openqa.selenium.remote.DesiredCapabilities;

public class 78644072022 {

public static void main(String[] args) {

    System.setProperty("webdriver.gecko.driver", "C:\\Utility\\BrowserDrivers\\geckodriver.exe");
    DesiredCapabilities dc = DesiredCapabilities.firefox();
    dc.setCapability("marionette", true);
    WebDriver driver =  new FirefoxDriver(dc);
    driver.manage().window().maximize();
    driver.get("https://www.shareinvestor.com/my");
    WebElement login_button = driver.findElement(By.xpath("//div[@id='sic_loginContainer']/div/div[@class='sic_logIn-bg']/a"));

    //Java Click
    login_button.click();

    //JavascriptExecutor Click
    /*JavascriptExecutor jse = (JavascriptExecutor)driver;
    jse.executeScript("arguments[0].click();", login_button);*/

    //Actions Click
    /*Actions act = new Actions(driver);
    act.moveToElement(login_button).click().build().perform();*/

    driver.findElement(By.xpath("//input[@id='sic_login_header_username']")).sendKeys("debanjan");
    driver.findElement(By.xpath("//input[@id='sic_login_header_password']")).sendKeys("5786");
    driver.findElement(By.xpath("//input[@id='sic_login_submit']")).click();
}
}

I am looking for a Java Solution with geckodriver to overcome the error JavaScript error:TypeError: document.getElementById(...) is null

In one of the SO threads I saw a solution like:

You need to do a null check in updateHTML like this:

function updateHTML(elmId, value) {
  var elem = document.getElementById(elmId);
  if(typeof elem !== 'undefined' && elem !== null) {
    document.getElementById(elmId).innerHTML = value;
  }
}

Can we implement this? Any suggestions & pointers will be helpful.

Answer 1

I am looking for a Java Solution with geckodriver to overcome the error JavaScript error:TypeError: document.getElementById(...) is null

To answer your question, I don't believe there's anything you can do to "fix" this via Java/Selenium. This is a JavaScript error, which originates from the website that you are trying to access. You might want to consider contacting their support team, maybe one of their developers can look at this issue.

---

Instead of clicking on the login button, maybe consider navigating directly to the login page?

driver.get("https://www.shareinvestor.com/user/login.html");

Answer 2

Firstly, Those javascript errors are not triggered due to any of the selenium code. Ofcourse, the timeout has been triggered by the selenium(will discuss on this on a later point).

You will get that javascript error irrespective of any kind of browser you launch the URL with. But in case of the gecko, you are notified in the eclipse console with but not in case of Chrome. If you need to see the java script error in chrome, just launch the url in chrome and go to devtools/console(F12). You can also see the same in firefox console too.

• Chrome Img:

Secondly, We are getting timeout exception because the site is really taking too much time to load. I have been waited for 7 minutes and the page is still loading even now. Selenium won't executes its script unless the page has been completely launched. As a result we are getting timeout exception(not sure about the default page launch time permitted). I thought of bypassing directly to the login page ("https://www.shareinvestor.com/user/login.html") and that's also not taking any finite time to load completely.

Intepolation: Those java script errors are not an issue for automation But those page loads are really. This site doesn't seems like a good candidate for automation with this issue.

Update1: otherwise we can also stop the page loading via another thread like sending "esc" key sequence using Action class.

Update2: I tried the same code today and it works fine. Below is the code snippet that i have tried(There is no change at all)

 public static void main(String[] args) throws InterruptedException 
{
    System.setProperty("webdriver.gecko.driver", "F:\\Softwares\\Selenium\\Webdriver\\geckodriver.exe");
    WebDriver driver = new FirefoxDriver();
    driver.manage().window().maximize();
    driver.manage().timeouts().implicitlyWait(5, TimeUnit.SECONDS);
    driver.get("https://www.shareinvestor.com/my");
    WebElement login_button = driver.findElement(By.xpath("//div[@id='sic_loginContainer']/div/div[@class='sic_logIn-bg']/a"));

    //Java Click
    login_button.click();
    System.out.println("In Login PAge");

    driver.findElement(By.xpath("//input[@id='sic_login_header_username']")).sendKeys("debanjan");
    driver.findElement(By.xpath("//input[@id='sic_login_header_password']")).sendKeys("5786");
    System.out.println("Entered password");
    driver.findElement(By.xpath("//input[@id='sic_login_submit']")).click();
}

Selenium version - 3.4.0

Gecko driver - v0.16.1(mine is 32 bit)

Mozilla - 51.0.1 (Update=>It's working on 53.02 also)

Hope this helps you. Thanks.

Answer 3

I think I've managed to find what is causing this uproar in your script. I inspected your HTML and it seems javascript method function showRemaining() is causing this problem; because showRemaining() contains this statement

    document.getElementById('countdown').innerHTML = '';

where it tries to set innerHTML attribute for element having id as countdown to ''.

But countdown doesn't exist anywhere on your web page hence the error.

TypeError: document.getElementById(...) is null

and somehow selenium isn't able to look past this error. So I think getting it fixed from developers should help you.

UPDATE :

Basically you need to wait all elements to load using implicit wait, once all elements are loaded, then your Javascript error gets resolved and ultimately interaction with the webpage is not hindered by that javascript error:

        driver.get("https://www.shareinvestor.com/my");

        driver.manage().timeouts().implicitlyWait(20, TimeUnit.SECONDS);
        driver.manage().window().maximize();

        /*String script = "function updateHTML(countdown, value) {" +
                "var elem = document.getElementById('countdown');"+
                "if(typeof elem !== 'undefined' && elem !== null) {"+
                " document.getElementById('countdown').innerHTML = value;"+
                "}"+
                "}";

        ((JavascriptExecutor) driver).executeScript(script);*/

        WebElement login_button = driver.findElement(By.xpath("//div[@id='sic_loginContainer']/div/div[@class='sic_logIn-bg']/a"));

        login_button.click();

        driver.findElement(By.xpath("//input[@id='sic_login_header_username']")).sendKeys("debanjan");
        driver.findElement(By.xpath("//input[@id='sic_login_header_password']")).sendKeys("5786");
        driver.findElement(By.xpath("//input[@id='sic_login_submit']")).click();

Answer 4

It looks like the XPath for the login_button is incorrect:

By.xpath("//div[@id='sic_loginContainer']/div/div[@class='sic_logIn-bg']/a");

Should be:

By.xpath("//div[@id='sic_loginContainer']/div/div/div[@class='sic_logIn-bg']/a");

---

This may explain the TimeoutException, since Selenium is unable to locate the non-existent element. EDIT: My mistake, you should see a NoSuchElementException if the element cannot be located.

As for the JavaScript error, unless you were trying to use Selenium to access the web element being manipulated by the JavaScript code (id='countdown' element), Selenium should just ignore the error. Otherwise, there's the potential for other Selenium errors, such as StaleElementReferenceException, but that does not seem to be the case.

Answer 5

This is due to your application HTML page using the async javascript and it uses some variable reference that is not available at the time of execution. we had same issue and asked developer to follow some best practices for javascript in the HTML like putting the script tag at end of the page. I have checked the HTML page source of the site and it contains many script tags in between the code. This will block the DOM rendering. Better, ask the developer to follow some best practices to include script tag in HTML. you refer the link Where should I put <script> tags in HTML markup? for the best practices.