Why does targeting shadow dom elements fail at the 5th element?

Question

Recently asked how to target elements in chrome settings here: How to edit chromes search and homepage with selenium/python?

I was told I should use 'shadow dom' elements, so I went ahead and figured out how to do that.

I was able to successfully target the search field on chrome's download page using shadow dom inception, but when applying almost identical logic/code to target the to open a specific page or set of pages on chrome://settings/ , python returned an error

no such element: Unable to locate element: {"method":"css selector","selector":"settings-on-startup-page"}

How do I solve the problem?

---

Eduard Florinescu commented: "when you usually have also dynamic content and more than 3 shadow elements one into another it makes impossible automation." here: How to handle elements inside Shadow DOM from Selenium

I'm hoping someone can explain the limitation or workaround here?

HERE IS EXAMPLE CODE THAT WORKS FOR TARGETING THE SEARCH FIELD ON CHROMES DOWNLOAD PAGE

import selenium
from selenium import webdriver
driver = webdriver.Chrome("C:/Users/John/Desktop/Documents/selenium/webdrivers/chromedriver.exe")

#define inception function
def expand_shadow_element(element):
  shadow_root = driver.execute_script('return arguments[0].shadowRoot', element)
  return shadow_root


#start doing inception
driver.get("chrome://downloads")
root1 = driver.find_element_by_tag_name('downloads-manager')
shadow_root1 = expand_shadow_element(root1)

root2 = shadow_root1.find_element_by_tag_name('downloads-toolbar')
shadow_root2 = expand_shadow_element(root2)

root3 = shadow_root2.find_element_by_tag_name('cr-toolbar')
shadow_root3 = expand_shadow_element(root3)

root4 = shadow_root3.find_element_by_css_selector("cr-toolbar-search-field")
shadow_root4 = expand_shadow_element(root4)

root5 = shadow_root4.find_element_by_id("searchInput")

root5.send_keys('test')

HERE IS EXAMPLE CODE THAT DOE NOT WORK FOR TARGETING THE OPEN A SPECIFIC PAGE OR SET OF PAGES RADIO BUTTON.

from selenium import webdriver
driver = webdriver.Chrome("C:/Users/John/Desktop/Documents/selenium/webdrivers/chromedriver.exe")


#define inception function
def expand_shadow_element(element):
  shadow_root = driver.execute_script('return arguments[0].shadowRoot', element)
  return shadow_root


#start doing inception
driver.get("chrome://settings/")
root1 = driver.find_element_by_tag_name('settings-ui')
shadow_root1 = expand_shadow_element(root1)

root2 = shadow_root1.find_element_by_tag_name('settings-main')
shadow_root2 = expand_shadow_element(root2)

root3 = shadow_root2.find_element_by_tag_name('settings-basic-page')
shadow_root3 = expand_shadow_element(root3)

root4 = shadow_root3.find_element_by_css_selector("settings-section")
shadow_root4 = expand_shadow_element(root4)

root5 = shadow_root4.find_element_by_css_selector("settings-on-startup-page")
shadow_root5 = expand_shadow_element(root5)

root6 = shadow_root5.find_element_by_css_selector("settings-radio-group")
shadow_root6 = expand_shadow_element(root6)

root7 = shadow_root6.find_element_by_name("4")

root7.click()

AS YOU CAN SEE, BOTH CODE SNIPPETS ARE ALMOST IDENTICAL IN SYNTAX AND STRUCTURE, YET THE SECOND DOESN'T WORK.

C:\Users\John\Desktop\Documents\selenium\projects\startpage_domain_test\venv\Scripts\python.exe C:/Users/John/Desktop/Documents/selenium/projects/startpage_domain_test/startpage_domain_test.py
Traceback (most recent call last):
  File "C:/Users/John/Desktop/Documents/selenium/projects/startpage_domain_test/startpage_domain_test.py", line 26, in <module>
    root5 = shadow_root4.find_element_by_css_selector("settings-on-startup-page")
  File "C:\Python37-32\lib\site-packages\selenium\webdriver\remote\webelement.py", line 430, in find_element_by_css_selector
    return self.find_element(by=By.CSS_SELECTOR, value=css_selector)
  File "C:\Python37-32\lib\site-packages\selenium\webdriver\remote\webelement.py", line 659, in find_element
    {"using": by, "value": value})['value']
  File "C:\Python37-32\lib\site-packages\selenium\webdriver\remote\webelement.py", line 633, in _execute
    return self._parent.execute(command, params)
  File "C:\Python37-32\lib\site-packages\selenium\webdriver\remote\webdriver.py", line 321, in execute
    self.error_handler.check_response(response)
  File "C:\Python37-32\lib\site-packages\selenium\webdriver\remote\errorhandler.py", line 242, in check_response
    raise exception_class(message, screen, stacktrace)
selenium.common.exceptions.NoSuchElementException: Message: no such element: Unable to locate element: {"method":"css selector","selector":"settings-on-startup-page"}
  (Session info: chrome=75.0.3770.142)


Process finished with exit code 1

Answer 1

It looks like you just have to recursively traverse the elements with shadowRoots.

Try it like this:

def find_in_shadow_dom(css):
  return driver.execute_async_script("""
    const traverse = e => {
      let el
      if(el = e.querySelector('""" + css + """')){
        arguments[0](el)
      }
      [...e.querySelectorAll('*')].filter(e => e.shadowRoot).map(e => traverse(e.shadowRoot))
    }
    [...document.querySelectorAll('*')].filter(e => e.shadowRoot).map(e => traverse(e.shadowRoot))
    arguments[0](null)
  """)

input = find_in_shadow_dom('#searchInput')
input.send_keys('testing')

Edit: Notes for settings page..

control = find_in_shadow_dom('[label="Open a specific page or set of pages"]')
# don't forget to scroll into view
driver.execute_script("arguments[0].scrollIntoView(true)", control)
control.click()

Answer 2

AFAIK, there is no such limit to access the shadow element. You should be good to access them until you provide the correct shadow tree elements.

Refer to answer here for detailed explanation. It's informative and detailed..

Here is the js that you can pass and return the element

input = driver.execute_script("return document.querySelector('downloads-manager').shadowRoot.querySelector('downloads-toolbar#toolbar').shadowRoot.querySelector('cr-toolbar#toolbar').shadowRoot.querySelector('cr-toolbar-search-field#search').shadowRoot.querySelector('input#searchInput')")
input.send_keys('testing')

Screenshot: