0

I am trying to web scrape using Java and Selenium but for one part of what I need to web scrape, it requires me to access a shadow DOM element. I am using Javascript executor to access it and I am entering the querySelector snippet needed to retrieve what I am looking for but I am getting a "Cannot read properties of null (reading 'ShadowRoot')" error. Is anyone familiar with this and how to solve it?

Here is my code snippet:

 String pagePdfUrl = (String) js.executeScript("document.querySelector('pdf-viewer').shadowRoot.getElementById('content').querySelector('embed').getAttribute('original-url')");

js is the JavascriptExectuor variable.

Thank you!

NoobCoder
  • 35
  • 1
  • 1
  • 6
  • No way for use to tell what you are doing right or wrong. You've described it as if you've done everything correctly. The property is named `shadowRoot` rather than `ShadowRoot` if that makes a difference. – Randy Casburn Dec 06 '21 at 02:46
  • My apologies. I have just added a code snippet. Please let me know if you need any more information – NoobCoder Dec 06 '21 at 02:54
  • Try putting `return ` between `"` and `document`. – Randy Casburn Dec 06 '21 at 02:56
  • [Reference](https://www.selenium.dev/selenium/docs/api/java/org/openqa/selenium/JavascriptExecutor.html#executeScript(java.lang.String,java.lang.Object...)) – Randy Casburn Dec 06 '21 at 02:57
  • Unfortunately that didn't work either – NoobCoder Dec 06 '21 at 03:05
  • the only thing left is your selector. It is returning `null` – Randy Casburn Dec 06 '21 at 03:06
  • I am testing this by going to the console of the webpage I'm accessing and entering this selector. It does not return null – NoobCoder Dec 06 '21 at 03:10
  • I believe you. At the time you call `executeScript()`, the DOM element(s) your are attempting to find are not there - your call returns null. When you do it in the browser console the entire DOM has been constructed. I don't know if these are dynamic elements that are injected or what. But that, I believe is the issue. – Randy Casburn Dec 06 '21 at 03:14
  • They actually are dynamic elements I believe. Is there a way to work around that? – NoobCoder Dec 06 '21 at 03:23
  • You can use `WebDriverWait(driver, delay).until(...)` with an xPath selector. I would stick with xPath as there doesn't seem to be any reason not to. It will reduce the complexity of your test code. – Randy Casburn Dec 06 '21 at 03:49
  • You can't access shadow DOM elements using xPath, can you? Furthermore, can WebDriverWait.until() work with querySelectors? – NoobCoder Dec 06 '21 at 03:58
  • Nope..., you could use `sleep()` I suppose. – Randy Casburn Dec 06 '21 at 04:00

3 Answers3

1

While using querySelector() to read a ShadowRoot this error message...

Cannot read properties of null

...implies either of the following circumstances:

Solution

In case of the first and second scenario, you won't be able to access the ShadowRoot. Where as in the third case you need to wait for sometime e.g. time.sleep(3) before you attempt to access the ShadowRoot and it's elements.

undetected Selenium
  • 183,867
  • 41
  • 278
  • 352
0

In Selenium 4.0, for Chromium versions 96+ you can use the getShadowRoot() method and avoid the JavaScript entirely.

A working example:

driver.get("http://watir.com/examples/shadow_dom.html");
WebElement shadowHost = driver.findElement(By.cssSelector("#shadow_host"));
SearchContext shadowRoot = shadowHost.getShadowRoot();
WebElement shadowContent = shadowRoot.findElement(By.cssSelector("#shadow_content"));

Source: https://titusfortner.com/2021/11/22/shadow-dom-selenium.html

titusfortner
  • 4,099
  • 2
  • 15
  • 29
0

Thread.sleep(10); is working fine for me.

Still i see same kind of error for another shadow element

sometimes shadow DOM querySelector('q2-input'); might have changed its location.

so if error still persists check the perfect JSPath and validate in console

Ramanujam RJ
  • 1