1

I am trying to use selenium to try and scrape this site. (https://suttacentral.net/mn86/en/sujato)

  1. get the element that represents the 'views' button at the upper right corner of the page below the search icon

  2. click on it

When I click on the button and inspect it, this is what I found

<mwc-icon-button class="white-icon toolButtons" id="btnTools" title="View options" slot="actionItems" hidden="" style="display: inherit;"> <!--?lit$530841421$--><svg class="visibility icon" xmlns="http://www.w3.org/2000/svg" height="24" viewBox="0 0 24 24" width="24"><path d="M0 0h24v24H0z" fill="none"></path><path d="M12 4.5C7 4.5 2.73 7.61 1 12c1.73 4.39 6 7.5 11 7.5s9.27-3.11 11-7.5c-1.73-4.39-6-7.5-11-7.5zM12 17c-2.76 0-5-2.24-5-5s2.24-5 5-5 5 2.24 5 5-2.24 5-5 5zm0-8c-1.66 0-3 1.34-3 3s1.34 3 3 3 3-1.34 3-3-1.34-3-3-3z"></path></svg> </mwc-icon-button>

So I thought I could retrieve it with the id 'btnTools'

This is the code I have tried:

from selenium import webdriver
import time
from selenium.webdriver.chrome.options import Options

PATH = "chromedriver"
driver = webdriver.Chrome(PATH)

driver.get("https://suttacentral.net/mn86/en/sujato")
time.sleep(10)
driver.find_element_by_id("btnTools")

and this is the error:

Traceback (most recent call last):
  File "/Users/nn/PycharmProjects/1palihelper/muck.py", line 12, in <module>
    driver.find_element_by_id("btnTools")
  File "/Users/nn/.virtualenvs/1palihelper/lib/python3.6/site-packages/selenium/webdriver/remote/webdriver.py", line 360, in find_element_by_id
    return self.find_element(by=By.ID, value=id_)
  File "/Users/nn/.virtualenvs/1palihelper/lib/python3.6/site-packages/selenium/webdriver/remote/webdriver.py", line 978, in find_element
    'value': value})['value']
  File "/Users/nn/.virtualenvs/1palihelper/lib/python3.6/site-packages/selenium/webdriver/remote/webdriver.py", line 321, in execute
    self.error_handler.check_response(response)
  File "/Users/nn/.virtualenvs/1palihelper/lib/python3.6/site-packages/selenium/webdriver/remote/errorhandler.py", line 242, in check_response
    raise exception_class(message, screen, stacktrace)
selenium.common.exceptions.NoSuchElementException: Message: no such element: Unable to locate element: {"method":"css selector","selector":"[id="btnTools"]"}
  (Session info: chrome=93.0.4577.63)

I have also noticed that when I click 'Developer Tools' and then 'Elements', the code I see there is different to what driver.page_source returns. Can someone please point in the right direction? driver.page_source does not return anything in the body.

Thanks Alot

  • 1
    Does this answer your question? [Does anybody know how to identify shadow dom web elements using selenium webdriver?](https://stackoverflow.com/questions/36141681/does-anybody-know-how-to-identify-shadow-dom-web-elements-using-selenium-webdriv) – Wiggy A. Sep 07 '21 at 05:01

0 Answers0