Dynamic JS content not loading when testing with Selenium and Google Chrome headless browser in Java

Question

I'm trying to test a website I made as part of a Spring Boot project, and I'm using Selenium. I'm able to test basic stuff like loading the page and the title, but I'm struggling with the actual content of the page.

My page has the following section:

<div id="main">
    <div id="div_1"></div>
    <div id="div_2"></div>
    <div id="div_3"></div>
    <div id="div_4"></div>
</div>

The content is loaded from a JS script:

document.addEventListener("DOMContentLoaded", function(event) {
    populateDivs()
})

I'm initialising the WebDriver with the following options gathered from similar questions (other Selenium tests continue passing, so I don't think the options are conflicting):

final ChromeOptions options = new ChromeOptions();
options.addArguments(
    "--headless",
    "--nogpu",
    "--disable-gpu",
    "--enable-javascript",
    "--no-sandbox",
    "--disable-extensions",
    "--disable-blink-features=AutomationControlled",
    "--disable-features=NetworkService,NetworkServiceInProcess",
     "start-maximized",
     "disable-infobars"
);

I've also added a wait in my test case to give time for the content to load:

final WebDriverWait wait = new WebDriverWait(driver, Duration.ofMinutes(1L));
wait.until(ExpectedConditions.visibilityOfElementLocated(By.id("div_1_content")));

Nothing I try changes the fact that the no content is loaded even after waiting. Not really sure where to go from here - am I using Selenium incorrectly? Or should I load my content in a different way on the JS side?

For a little more context, here is how I am loading the page in Selenium:

// Create driver
final ChromeOptions options = new ChromeOptions();
options.setAcceptInsecureCerts(true);
options.addArguments("--headless=new");

final URL url = new URL(http://127.0.0.1:4444/wd/hub);
final RemoteWebDriver driver= new RemoteWebDriver(url, options);


// Load page
driver.navigate().to("https://127.0.0.1:81");


// Wait for dynamic content to load
final WebDriverWait wait = new WebDriverWait(driver, Duration.ofMinutes(1L));            wait.until(ExpectedConditions.visibilityOfElementLocated(By.id("div_1_content")));

Can you share your Selenium code? How are you launching the URL? If you launch the same URL manually is it working? Whats the error you are getting in console? Please post most about it. — Ahamed Abdul Rahman, Jun 23 '23 at 23:47
@AhamedAbdulRahman, I have updated the post with my Selenium code. I can launch the URL in Chrome manually and can see that the content is being populated correctly. I am getting no errors in console, it just looks like Selenium doesn't recognise any of the content created by my JS scripts. — zodac, Jun 28 '23 at 21:53
May I know why are you focusing on Headless mode? Did you try Headed mode? Did it worked? — Ahamed Abdul Rahman, Jun 29 '23 at 09:07
Headed mode meaning just not including the `--headless` option? Yes, I tried that first which had the same failing results. Unless there is some more explicit way of using headed mode? I'm using the Selenium docker image (https://hub.docker.com/r/selenium/standalone-chrome), not sure if that makes a difference? — zodac, Jun 29 '23 at 21:25
Did you try to launch the service locally and trigger the Selenium ChromeDriver script? Also please use driver.get() to launch the url instead of driver.navigate().to(). https://stackoverflow.com/questions/33865618/difference-between-webdriver-get-and-webdriver-navigate-to-in-the-case-of-ur/33868976#33868976 — Ahamed Abdul Rahman, Jul 01 '23 at 01:47

undetected Selenium · Answer 1 · 2023-06-29T21:52:11.763

You need to take care of a couple of things as follows:

The headless property is deprecated for quite sometime now. Between Chrome versions 96 & 108 it was --headless=chrome and Chrome version 109 onwards it's --headless=new

Note: What are the two headless modes?

--nogpu and --disable-gpu arguments have no more revelance in the modern day Chrome browser. So you can drop them.
Modern day browser clients won't function properly without Javascript and it's a mandatory requirement. So you need to drop --enable-javascript.
--no-sandbox isn't required unless you are facing issues executing your test as a root/admin user of the system and can be dropped.
--disable-extensions and disable-infobars arguments have no more revalance. So you can drop them.

Network Service

There are some services for networking. Those are meant to be oblivious to Chrome's features.

this only contains features that go over the network. e.g. no file loading, data URLs etc...

only the lowest-level of networking should be here. e.g. http, sockets, web sockets. Anything that is built on top of this should be in higher layers.

higher level web platform and browser features should be built outside of this code. Safe browsing, Service Worker, extensions, devtools etc... should not have hooks here. The only exception is when it's impossible for these features to function without some hooks in the network service. In that case, we add the minimal code required. Some examples included traffic shaping for devtools, CORB blocking, and CORS.

every PostTask, thread hop and process hop (IPC) should be counted carefully as they introduce delays which could harm this performance critical code.

NetworkContext and NetworkService are trusted interfaces that aren't meant to be sent to the renderer. Only the browser should have access to them.

This usecase

NetworkService and NetworkServiceInProcess are two important configuration and you shouldn't disable them. So you need to remove the argument:

"--disable-features=NetworkService,NetworkServiceInProcess"

Incorporating the above suggestions execute your testcase, you should be good to go.

tl; dr

Some useful documentations:

Update

As per your comment update that you still cannot see any of the dynamic content being loaded even after inducing WebDriverWait can be due to one of the numerous reasons as below:

The browser is transferring the control to the WebDriver even before document.readyState as complete is achieved. In that case you can add the following code block:
```
new WebDriverWait(driver, Duration.ofMinutes(1L)).until(d -> ((JavascriptExecutor)d).executeScript("return document.readyState").equals("complete"));
```
Additionally as the WebDriver is generally assumed to have a blocking API as it is an out-of-process library that instructs the browser what to do and the web platform having an intrinsically asynchronous nature, WebDriver does not track the active, real-time state of the DOM and most intermittent issues that arise from use of Selenium and WebDriver are connected to race conditions that occur between the browser and the user’s instructions. To be on the safer side you can add a method to ensure return jQuery.active == 0 as follows:
```
public void WaitForAjax2Complete() throws InterruptedException
{
    while (true)
    {
        if ((Boolean) ((JavascriptExecutor)driver).executeScript("return jQuery.active == 0")){
            break;
        }
        Thread.sleep(100);
    }
}
```
You also need to ensure that if the desired elements are within any iframe
Finaly, you also need to ensure that if the desired elements are within any #shadow-root

Thanks @undetected Selenium, appreciate you going through each of those options and confirming most were useless. :) I got rid of them all and only kept '--headless=new'. However, that doesn't seem to have changed anything, I still cannot see any of the dynamic content being loaded. — zodac, Jun 28 '23 at 21:43
@zodac Checkout the answer update and let me know the status. — undetected Selenium, Jun 29 '23 at 21:52
Thanks for the additional comments. However, doesn't seem like they have resolved things. :( I added jquery to my site then added both code snippets to the test, but the dynamic content is still not being found by Selenium. I've also not used any iframes, nor have I used any 'shadow' DOM elements. — zodac, Jun 30 '23 at 01:32
@zodac Do you have any antivirus installed on your localhost? — undetected Selenium, Jun 30 '23 at 19:59

score 0 · Answer 2 · answered Jun 28 '23 at 20:19

In your example code, you had

wait.until(ExpectedConditions.visibilityOfElementLocated(By.id("div_1_content")));

...which used By.id("div_1_content"), but there was no id with that value in your html:

<div id="main">
    <div id="div_1"></div>
    <div id="div_2"></div>
    <div id="div_3"></div>
    <div id="div_4"></div>
</div>

So instead of using div_1_content as the id, use div_1. Otherwise your element won't be found.

Also, use --headless=new instead of --headless (https://stackoverflow.com/a/73840130/7058266), so that headless mode loads just like regular Chrome.

Regarding document.addEventListener("DOMContentLoaded", function(event) - https://developer.mozilla.org/en-US/docs/Web/API/Window/DOMContentLoaded_event

The DOMContentLoaded event fires when the HTML document has been completely parsed, and all deferred scripts ( and ) have downloaded and executed. It doesn't wait for other things like images, subframes, and async scripts to finish loading... DOMContentLoaded does not wait for stylesheets to load... A different event, load, should be used only to detect a fully-loaded page.

Which means there's a possibility of the event listener firing before you want it to, unless you use this instead: addEventListener("load", (event) => {});

But changing the id from div_1_content to div_1 might be enough to solve your issue.

Thanks for the reply @Michael Mintz. The HTML I provided was the static content on my page - when the page loads the JS dynamically creates `div_1_content` and populates it with data from a REST call to the backend. My issue is that while I can load the page in the Chrome browser and the correct content appears, I cannot find it using Selenium. I did try using `addEventListener("load", (event) => {});` instead of `addEventListener("DOMContentLoaded", function(event){})`, and used `--headless=new` when creating the driver, but the dynamic content still does not load. — zodac, Jun 28 '23 at 21:47
Is the dynamic content generated in an iframe? If so, you need to switch to the iframe first before you can find elements inside it: https://www.selenium.dev/documentation/webdriver/interactions/frames/ — Michael Mintz, Jun 28 '23 at 21:50
No iframes, just `document.createElement()`s and `.append()`s. Something like this: ```newDiv = document.createElement("div"); newDiv.setAttribute("id", "innerdiv"+index); newDiv.setAttribute("class", "collapse");" ``` — zodac, Jun 28 '23 at 22:45

Dynamic JS content not loading when testing with Selenium and Google Chrome headless browser in Java

2 Answers2

Network Service

This usecase

tl; dr

Update