Unable to find element in nested shodow root in Selenium in python

Question

I have read similar answers but it doesnt seem to be working for me . I am trying to scrape product names from a website

driver = webdriver.Chrome(DRIVER_PATH,options=options)
website = 'https://bodyandmind.com/california/longbeachdispensary/longbeachmenu/?category=flower#inventory'
driver.get(website)
time.sleep(3)
confirmation = driver.find_element(By.CSS_SELECTOR,'button.age-gate-submit.age-gate-submit-yes').click()
time.sleep(3)

root1=driver.find_element(By.CSS_SELECTOR, "weave-ordering").shadow_root

root2=root1.find_element(By.CSS_SELECTOR, "weave-app").shadow_root

root3=root2.find_element(By.CSS_SELECTOR, "weave-inventory").shadow_root

root4=root3.find_element(By.CSS_SELECTOR, "weave-product-list").shadow_root

root5=root4.find_element(By.CSS_SELECTOR, "weave-category-section").shadow_root

root6=root5.find_element(By.CSS_SELECTOR, "weave-product").shadow_root

element=root6.find_element(By.CSS_SELECTOR, "[class='product-container']")

print(element.text)

Running it is giving me the following error :

Traceback (most recent call last):
  File "/Users/japneeshsingh/Desktop/try.py", line 29, in <module>
    root1=driver.find_element(By.CSS_SELECTOR, "weave-ordering").shadow_root
  File "/Users/japneeshsingh/opt/anaconda3/lib/python3.8/site-packages/selenium/webdriver/remote/webelement.py", line 252, in shadow_root
    return self._execute(Command.GET_SHADOW_ROOT)["value"]
  File "/Users/japneeshsingh/opt/anaconda3/lib/python3.8/site-packages/selenium/webdriver/remote/webelement.py", line 403, in _execute
    return self._parent.execute(command, params)
  File "/Users/japneeshsingh/opt/anaconda3/lib/python3.8/site-packages/selenium/webdriver/remote/webdriver.py", line 440, in execute
    self.error_handler.check_response(response)
  File "/Users/japneeshsingh/opt/anaconda3/lib/python3.8/site-packages/selenium/webdriver/remote/errorhandler.py", line 245, in check_response
    raise exception_class(message, screen, stacktrace)
selenium.common.exceptions.NoSuchShadowRootException: Message: no such shadow root

undetected Selenium · Answer 1 · 2023-06-28T23:52:42.770

The product names within multiple #shadow-root (open)

multiple-shadow-root

Solution

To extract the product names you need to use querySelectorAll() and you can use the following locator strategies:

driver.get("https://bodyandmind.com/california/longbeachdispensary/longbeachmenu/?category=flower#inventory")
WebDriverWait(driver, 20).until(EC.element_to_be_clickable((By.CSS_SELECTOR, "button.age-gate-submit.age-gate-submit-yes"))).click()
time.sleep(15)
elements = driver.execute_script("return document.querySelector('weave-ordering').shadowRoot.querySelector('weave-app').shadowRoot.querySelector('weave-inventory').shadowRoot.querySelector('weave-product-list').shadowRoot.querySelector('weave-category-section').shadowRoot.querySelectorAll('weave-product')")
for element in elements:
    ele = element.shadow_root
    e = ele.find_element(By.CSS_SELECTOR, "h2")
    print(e.text)

Console Output:

805 Glue
805 Glue
805 Glue
805 Glue Sugar Shake
Ajo Blanco
Alien Cookies
Animal Face
Artemis
BK Satellite
Banana Sherbet
Banana Sherbet
Biskante
Black Diamond Smalls
Black Diamond X
Black Triangle OG
Black Triangle OG
Blue Cookiez
Blue Dream
Blue Dream
Blue Dream
Blue Dream Sugar Shake
Blue Lotus
Blueberry Mac
Blueberry Skittles
Blueberry Skittles
Cadillac Rainbowz
Cali Berry
Cereal Milk Sugar Shake
Chauffeur
Cheetah Piss
Chem Dawg
Chem Dawg
Cherry Cheesecake
Cherry Gas
Cherry Gorilla
Cherry Popperz
Cherry Sababa
Cherry Zest #4
Cocolato
Crop Duster
DJ Short Blueberry
Diesel Tov
Double Diesel
Drip
Dutch Treat
Electric Blue
Fantasma
French Laundry
Fritter Glitter
Fruit Bubblegum
GMO Cookies Smalls
GMO-Sherb
Garlic Starship
Gelato
Gelato 41
Gelato Quin
Gelatti
Ghost OG Smalls
Goji Berry Runtz
Grape Ambrosia
Grape Cake
Grape Octane
Grapes Of Wrath
Gravitron
Grease Bucket #9
Grease Monkey
Gruntz
Hella Jelly
Highrise
Holy Moly!
Honey Lemon Hibiscus
Infused Baby Jeeter Grapefruit Romulan
Jelly Fish CBD
Kiwi Sorbet
Kosher Kush
Krypto Chronic
Kush Cake x Hybrid Kief Power Pack
Kush Mints
Kush Mints
Kush Mints
Kush Mints Sugar Shake
L'Chaim Lemon
Lava Cake Smalls
Lemon Cherry Gelato
Lemon Granita
Lemon z
Lilac Diesel
Lilac Diesel
MAC 1
MVP Cookies
Mac 1
Mac 1
Mac1
Mac1 x Hybrid Kief Power Pack
Mafia Funeral
Malibu Pure Kush
Mango Haze
Mango Haze
Mimosa
Modified Grapes
Moonana Wreck
Motor Breath Smalls
Mule Fuel
NF1
ORO

References

You can find a couple of relevant detailed discussions in:

score 1 · Accepted Answer · answered Jun 29 '23 at 05:24

Your issue is with the weave-product line. If you remove that line and shift your find_element() calls, then it works (assuming you wait for the page to fully load before searching):

root1=driver.find_element(By.CSS_SELECTOR, "weave-ordering").shadow_root
root2=root1.find_element(By.CSS_SELECTOR, "weave-app").shadow_root
root3=root2.find_element(By.CSS_SELECTOR, "weave-inventory").shadow_root
root4=root3.find_element(By.CSS_SELECTOR, "weave-product-list").shadow_root
root5=root4.find_element(By.CSS_SELECTOR, "weave-category-section").shadow_root
element=root5.find_element(By.CSS_SELECTOR, "[class='product-container']")

To continue and get individual product names, add the following:

all_products = element.find_elements(By.CSS_SELECTOR, "weave-product")
for product in all_products:
    print(product.text.split("\n")[0])

That outputs:

Pacific Stone
Pacific Stone
Pacific Stone
Maven Genetics
Mazel Tov
Fig Farms
Cream of The Crop
Alien Labs
Claybourne Co.
Claybourne Co.
Alien Labs
West Coast Trading Company
Maven Genetics
...

Unable to find element in nested shodow root in Selenium in python

2 Answers2

Solution

References