0

I am trying to use Selenium in order to learn different ways of web scraping.

When the code is executed Firefox starts and the "accept cookies" or what ever pops up. I am unable to locate the "accept" button when inspecting the page.

my code so far:

from selenium import webdriver
from selenium.webdriver.common.keys import Keys
from selenium.webdriver.common.by import By
from selenium.webdriver.chrome.service import Service
import pandas as pd
import time

PATH = "C:/Users/myuser/Desktop/Driver/geckodriver.exe"

driver = webdriver.Firefox(executable_path=PATH)


driver.maximize_window() # For maximizing window
driver.get("https://www.immonet.de/")


button_pos = driver.find_element(by=By.CLASS_NAME, value="sc-gsDKAQ fILFKg")
button_pos.click()
print(driver.title)



input = input()

I get the following error: Unable to locate element: .sc-gsDKAQ fILFKg

My thought was locating the button via the inspect tool as follows:

enter image description here

What am I missing or doing wrong? How would i find the right element?

Thanks! Pat

Trdlo
  • 45
  • 5

2 Answers2

1

First of all, to display this url,accepting the cookies is a must but to accept and click on the cookie button isn't a easy task because cookies button is under shadow root (open) selenium and webdriverWait can do nothing on shadow root,so to execute shadow root you need to apply JavaScript querySelector.

#To execute shadow root and accept cookies

driver.execute_script('''return document.querySelector('div#usercentrics-root').shadowRoot.querySelector('button[data-testid="uc-accept-all-button"]')''').click()
Md. Fazlul Hoque
  • 15,806
  • 5
  • 12
  • 32
  • So I understand that XPATH is not usable with shadow root open and therefore I need to use `driver.execute_script()`. When I use the chrome extension "SelectorsHub" I am identifiying the button as `document.querySelector(".sc-gsDKAQ.fILFKg")`. Can you explain to meh why you chose a different path? I just tried copying your code and run it, but it throws an error: "'NoneType' object has no attribute 'click'". I am sure I am missing some basic knowledge and am thankful for any advice – Trdlo Jun 13 '22 at 14:24
  • @Trdlo You can choose any one either class value that you you have selected or of mine. Did your selection work? – Md. Fazlul Hoque Jun 13 '22 at 14:44
  • unfortunately not. I tried the following `driver = webdriver.Firefox(executable_path=PATH) driver.maximize_window() # For maximizing window driver.get("https://www.immonet.de/") item = driver.execute_script('''return document.querySelector('div#usercentrics-root').shadowRoot.querySelector('button[data-testid="uc-accept-all-button"]')''') item.click()` But I keep getting the error : "'NoneType' object has no attribute 'click'" Edit: Sorry that this looks ugly formatted. Is there a way to post this more clearly in a comment or should I edit my original post? – Trdlo Jun 13 '22 at 14:57
  • I think, nothing wrong is happening here. You can a bit analyze from the same website, same type of problem's solution:https://stackoverflow.com/questions/71055919/how-to-interact-with-cookie-pop-within-shadow-root-open-using-selenium – Md. Fazlul Hoque Jun 13 '22 at 17:05
  • Oh wow, thanks for the link! This looks like the same problem. I will further investigate. Thanks for your time so far! – Trdlo Jun 13 '22 at 17:11
0

Class attribute in the html element can contain multiple classes separated by space. i.e. "sc-gsDKAQ fILFKg", contains two classes, sc-gsDKAQ and fILFKg.

You can user either but both are random and can be changed next time css is recompiled. I recommend to think of xpath using data-testid attribute

Yevhen B
  • 306
  • 1
  • 4