How to handle elements inside Shadow DOM from Selenium

Question

I want to automate file download completion checking in chromedriver. HTML of each entry in downloads list looks like

<a is="action-link" id="file-link" tabindex="0" role="link" href="http://fileSource" class="">DownloadedFile#1</a>

So I use following code to find target elements:

driver.get('chrome://downloads/')  # This page should be available for everyone who use Chrome browser
driver.find_elements_by_tag_name('a')

This returns empty list while there are 3 new downloads.

As I found out, only parent elements of #shadow-root (open) tag can be handled. So How can I find elements inside this #shadow-root element?

okay, then probably Css/Xpath remains as the means to access `driver.find_elements_by_css_selector(".[id='file-link']")` provides you some value? — Naman, May 23 '16 at 07:31
your statement returns `InvalidSelectorException`, `driver.find_elements_by_css_selector("[id='file-link']")` returns empty list — Andersson, May 23 '16 at 07:34
@Anderson : did you miss the `.` after " in `driver.find_elements_by_css_selector(".[id='file-link']")` ? — Naman, May 23 '16 at 07:35
just curious, why aren't you looking for the `progressContainer` or `` class? — Naman, May 23 '16 at 07:49
You can keep an eye on which file's progress is completed or which has a Pause/Resume button displayed against it. Does that not solve your purpose? — Naman, May 23 '16 at 07:56
Let us [continue this discussion in chat](http://chat.stackoverflow.com/rooms/112651/discussion-between-andersson-and-nullpointer). — Andersson, May 23 '16 at 08:03

Eduard Florinescu · Answer 1 · 2018-02-13T23:40:05.323

Sometimes the shadow root elements are nested and the second shadow root is not visible in document root, but is available in its parent accessed shadow root. I think is better to use the selenium selectors and inject the script just to take the shadow root:

def expand_shadow_element(element):
  shadow_root = driver.execute_script('return arguments[0].shadowRoot', element)
  return shadow_root

outer = expand_shadow_element(driver.find_element_by_css_selector("#test_button"))
inner = outer.find_element_by_id("inner_button")
inner.click()

To put this into perspective I just added a testable example with Chrome's download page, clicking the search button needs open 3 nested shadow root elements:

import selenium
from selenium import webdriver
driver = webdriver.Chrome()


def expand_shadow_element(element):
  shadow_root = driver.execute_script('return arguments[0].shadowRoot', element)
  return shadow_root

driver.get("chrome://downloads")
root1 = driver.find_element_by_tag_name('downloads-manager')
shadow_root1 = expand_shadow_element(root1)

root2 = shadow_root1.find_element_by_css_selector('downloads-toolbar')
shadow_root2 = expand_shadow_element(root2)

root3 = shadow_root2.find_element_by_css_selector('cr-search-field')
shadow_root3 = expand_shadow_element(root3)

search_button = shadow_root3.find_element_by_css_selector("#search-button")
search_button.click()

Doing the same approach suggested in the other answers has the drawback that it hard-codes the queries, is less readable and you cannot use the intermediary selections for other actions:

search_button = driver.execute_script('return document.querySelector("downloads-manager").shadowRoot.querySelector("downloads-toolbar").shadowRoot.querySelector("cr-search-field").shadowRoot.querySelector("#search-button")')
search_button.click()

later edit:

I recently try to access the content settings(see code below) and it has more than one shadow root elements imbricated now you cannot access one without first expanding the other, when you usually have also dynamic content and more than 3 shadow elements one into another it makes impossible automation. The answer above use to work a few time ago but is enough for just one element to change position and you need to always go with inspect element an ho up the tree an see if it is in a shadow root, automation nightmare.

Not only was hard to find just the content settings due to the shadowroots and dynamic change when you find the button is not clickable at this point.

driver = webdriver.Chrome()


def expand_shadow_element(element):
  shadow_root = driver.execute_script('return arguments[0].shadowRoot', element)
  return shadow_root

driver.get("chrome://settings")
root1 = driver.find_element_by_tag_name('settings-ui')
shadow_root1 = expand_shadow_element(root1)

root2 = shadow_root1.find_element_by_css_selector('[page-name="Settings"]')
shadow_root2 = expand_shadow_element(root2)

root3 = shadow_root2.find_element_by_id('search')
shadow_root3 = expand_shadow_element(root3)

search_button = shadow_root3.find_element_by_id("searchTerm")
search_button.click()

text_area = shadow_root3.find_element_by_id('searchInput')
text_area.send_keys("content settings")

root0 = shadow_root1.find_element_by_id('main')
shadow_root0_s = expand_shadow_element(root0)


root1_p = shadow_root0_s.find_element_by_css_selector('settings-basic-page')
shadow_root1_p = expand_shadow_element(root1_p)


root1_s = shadow_root1_p.find_element_by_css_selector('settings-privacy-page')
shadow_root1_s = expand_shadow_element(root1_s)

content_settings_div = shadow_root1_s.find_element_by_css_selector('#site-settings-subpage-trigger')
content_settings = content_settings_div.find_element_by_css_selector("button")
content_settings.click()

Hi Eduard I'm late to the party. I tried to use your code but it seems that `shadow_root1` does not have the `find_element_by_whatever` method. Did I do anything wrong? Bascially I have `root1 = driver.find_element_by_tag_name('input')` and then `shadowRoot1 = ExpandShadowElement(root1)` — Nicholas Humphrey, Nov 30 '18 at 16:02
They keep changing it and haven't got he time to look at it and update — Eduard Florinescu, Nov 30 '18 at 23:30
Ah, thanks! Actually I found out I don't need to parse the shadow DOM, managed to log in without touching them, dunno why... — Nicholas Humphrey, Nov 30 '18 at 23:40

score 8 · Answer 2 · answered Jul 07 '21 at 12:42

There is also ready to use pyshadow pip module, which worked in my case, below example:

from pyshadow.main import Shadow
from selenium import webdriver

driver = webdriver.Chrome('chromedriver.exe')
shadow = Shadow(driver)
element = shadow.find_element("#Selector_level1")
element1 = shadow.find_element("#Selector_level2")
element2 = shadow.find_element("#Selector_level3")
element3 = shadow.find_element("#Selector_level4")
element4 = shadow.find_element("#Selector_level5")
element5 = shadow.find_element('#control-button') #target selector
element5.click()

I found pyshadow is only working on Chrome. In my case, it doesn't work on Firefox or Safari. — Zac, May 23 '22 at 07:25

Supersharp · Answer 3 · 2016-05-25T13:58:37.133

2

You can use the driver.executeScript() method to access the HTML elements and JavaScript objects in your web page.

In the exemple below, executeScript will return in a Promise the Node List of all <a> elements present in the Shadow tree of element which id is host. Then you can perform you assertion test:

it( 'check shadow root content', function () 
{
    return driver.executeScript( function ()
    {
        return host.shadowRoot.querySelectorAll( 'a' ).then( function ( n ) 
        {
            return expect( n ).to.have.length( 3 )
        }
    } )
} )

Note: I don't know Python so I've used the JavaScript syntax but it should work the same way.

edited May 25 '16 at 13:58

answered May 23 '16 at 10:10

Supersharp

29,002
9
92
134

I have no idea about what this code means :) Also I've never seen `=>` symbol in `JS` What it used for?... can anyone "translate" this code? – Andersson May 25 '16 at 11:36
() => is a lambda expression / inline function syntax. I updated my anwer to use a standard function declaration. – Supersharp May 25 '16 at 13:44

Bjc · Answer 4 · 2019-05-20T20:15:46.940

I would add this as a comment but I don't have enough reputation points--

The answers by Eduard Florinescu works well with the caveat that once you're inside a shadowRoot, you only have the selenium methods available that correspond to the available JS methods--mainly select by id.

To get around this I wrote a longer JS function in a python string and used native JS methods and attributes (find by id, children + indexing etc.) to get the element I ultimately needed.

You can use this method to also access shadowRoots of child elements and so on when the JS string is run using driver.execute_script()

Dariopnc · Answer 5 · 2022-05-13T16:21:36.880

With selenium 4.1 there's a new attribute shadow_root for the WebElement class.

From the docs:

Returns a shadow root of the element if there is one or an error. Only works from Chromium 96 onwards. Previous versions of Chromium based browsers will throw an assertion exception.

Returns:

ShadowRoot object or
NoSuchShadowRoot - if no shadow root was attached to element

A ShadowRoot object has the methods find_element and find_elements but they're currently limited to:

By.ID
By.CSS_SELECTOR
By.NAME
By.CLASS_NAME

Shadow roots and explicit waits

You can also combine that with WebdriverWait and expected_conditions to obtain a decent behaviour. The only caveat is that you must use EC that accept WebElement objects. At the moment it's just one of the following ones:

element_selection_state_to_be
element_to_be_clickable
element_to_be_selected
invisibility_of_element
staleness_of
visibility_of

Example

e.g. borrowing the example from eduard-florinescu

from selenium.webdriver.support.ui import WebDriverWait

driver = webdriver.Chrome()
timeout = 10

driver.get("chrome://settings")
root1 = driver.find_element_by_tag_name('settings-ui')
shadow_root1 = root1.shadow_root

root2 = WebDriverWait(driver, timeout).until(EC.visibility_of(shadow_root1.find_element(by=By.CSS_SELECTOR, value='[page-name="Settings"]')))
shadow_root2 = root2.shadow_root

root3 = WebDriverWait(driver, timeout).until(EC.visibility_of(shadow_root2.find_element(by=By.ID, value='search')))
shadow_root3 = root3.shadow_root

search_button = WebDriverWait(driver, timeout).until(EC.visibility_of(shadow_root3.find_element(by=By.ID, value="searchTerm")))
search_button.click()

text_area = WebDriverWait(driver, timeout).until(EC.visibility_of(shadow_root3.find_element(by=By.ID, value='searchInput')))
text_area.send_keys("content settings")

root0 = WebDriverWait(driver, timeout).until(EC.visibility_of(shadow_root1.find_element(by=By.ID, value='main')))
shadow_root0_s = root0.shadow_root


root1_p = WebDriverWait(driver, timeout).until(EC.visibility_of(shadow_root0_s.find_element(by=By.CSS_SELECTOR, value='settings-basic-page')))
shadow_root1_p = root1_p.shadow_root


root1_s = WebDriverWait(driver, timeout).until(EC.visibility_of(shadow_root1_p.find_element(by=By.CSS_SELECTOR, value='settings-privacy-page')))
shadow_root1_s = root1_s.shadow_root

content_settings_div = WebDriverWait(driver, timeout).until(EC.visibility_of(shadow_root1_s.find_element(by=By.CSS_SELECTOR, value='#site-settings-subpage-trigger')))
content_settings = WebDriverWait(driver, timeout).until(EC.visibility_of(content_settings_div.find_element(by=By.CSS_SELECTOR, value="button")))
content_settings.click()

score 0 · Answer 6 · answered Nov 25 '21 at 00:59

I originally implemented Eduard's solution just slightly modified as a loop for simplicity. But when Chrome updated to 96.0.4664.45 selenium started returning a dict instead of a WebElement when calling 'return arguments[0].shadowRoot'.

I did a little hacking around and found out I could get Selenium to return a WebElement by calling return arguments[0].shadowRoot.querySelector("tag").

Here's what my final solution ended up looking like:

def get_balance_element(self):
        # Loop through nested shadow root tags
        tags = [
            "tag2",
            "tag3",
            "tag4",
            "tag5",
            ]

        root = self.driver.find_element_by_tag_name("tag1")

        for tag in tags:
            root = self.expand_shadow_element(root, tag)

        # Finally there.  GOLD!

        return [root]

def expand_shadow_element(self, element, tag):
    shadow_root = self.driver.execute_script(
        f'return arguments[0].shadowRoot.querySelector("{tag}")', element)
    return shadow_root

Clean and simple, works for me.

Also, I could only get this working Selenium 3.141.0. 4.1 has a half baked shadow DOM implementation that just manages to break everything.

Chrome 96+ is designed to work with the new `shadow_dom` property in Python Selenium 4.1. I also have a hack for Selenium 3 here: https://titusfortner.com/2021/11/22/shadow-dom-selenium.html — titusfortner, Dec 13 '21 at 05:16

score 0 · Answer 7 · answered Aug 12 '22 at 11:31

The downloaded items by google-chrome are within multiple #shadow-root (open).

chrome_downloads

Solution

To extract the contents of the table you have to use shadowRoot.querySelector() and you can use the following locator strategy:

Code Block:

driver = webdriver.Chrome(service=s, options=options)
driver.execute("get", {'url': 'chrome://downloads/'})
time.sleep(5)
download = driver.execute_script("""return document.querySelector('downloads-manager').shadowRoot.querySelector('downloads-item').shadowRoot.querySelector('a#file-link')""")
print(download.text)

How to handle elements inside Shadow DOM from Selenium

7 Answers7

later edit:

From the docs:

Shadow roots and explicit waits

Example

Solution

Linked

Related