3

I am trying to parse this website using selenium but I fail to find the buttons of the cookie popup which I need to confirm to proceed. I know that I first need to load the page and then wait for the cookie popup to appear, although that should be well handled by the sleep function.

The html of the buttons inside the popup looks like this (retrieved via firefox F12). I am trying to simply click the "OK" button:

<button role="button" data-testid="uc-customize-button" style="border: 1px solid rgb(0, 139, 2);" class="sc-gsDKAQ hWcdhQ">Einstellungen oder ablehnen</button>
<div class="sc-bBHxTw foBPAO"></div>
<button role="button" data-testid="uc-accept-all-button" style="border: 1px solid rgb(0, 139, 2);" class="sc-gsDKAQ fILFKg">OK</button>

I have tried to simply find the buttons via the xpath of the "OK" button:

from time import sleep
from selenium import webdriver
from selenium.webdriver.common.by import By

# adapting: https://stackoverflow.com/questions/64032271/handling-accept-cookies-popup-with-selenium-in-python

BASE_URL = "https://www.immowelt.at/suche/wohnungen/mieten"

driver = webdriver.Firefox()
driver.get(BASE_URL)
sleep(10)

# cannot locate by XPATH
driver.find_element(
    By.XPATH, "/div/div/div[2]/div/div[2]/div/div/div/button[2]"
).click()

# weird thing is, I also cannot retrieve any of the buttons (although I know they exist)
buttons = driver.find_elements(By.TAG_NAME, "button")

Any ideas why I can't find the button?

Guillaume G
  • 313
  • 1
  • 2
  • 15
geopanda1
  • 67
  • 6
  • Use the following Xpath: //*[@id="uc-center-container"]/div[2]/div/div/div/button[2] – Wudfulstan Dec 21 '22 at 09:13
  • 1
    Or just use driver.execute_script("""document.querySelector("#usercentrics-root").shadowRoot.querySelector("#uc-center-container > div.sc-ikJyIC.gpszuz > div > div > div > button.sc-gsDKAQ.fILFKg").click()""") – Wudfulstan Dec 21 '22 at 09:15
  • @Wudfulstan using "driver.execute_script..." works fine, using the solution based on xpath does not find the element – geopanda1 Dec 21 '22 at 09:48

1 Answers1

3

By inspecting the HTML we can see that the button is inside a shadow root.

enter image description here

So in order to be able to interact with the button we must first select the parent of the shadow root, which is the div element with id=usercentrics-root, and then we can select the button and click it:

driver.execute_script('''return document.querySelector("#usercentrics-root").shadowRoot.querySelector("button[data-testid='uc-accept-all-button']")''').click()

Alternatively, a more human readable code:

shadow_parent = driver.find_element(By.CSS_SELECTOR, '#usercentrics-root')
outer = driver.execute_script('return arguments[0].shadowRoot', shadow_parent)
outer.find_element(By.CSS_SELECTOR, "button[data-testid='uc-accept-all-button']").click()
sound wave
  • 3,191
  • 3
  • 11
  • 29
  • 1
    "driver.execute..." worked just fine using the Firefox driver, for the more human readable version, I had to change `arguments[0].shadowRoot` to `arguments[0].shadowRoot.children` and consequently `outer` to `outer[0]` in the last line, due to a `JavascriptException: Message: Cyclic object value` which seems related to [this](https://stackoverflow.com/questions/67133483/accessing-shadowroot-via-selenium-in-firefox-returns-javascriptexception-cyclic) – geopanda1 Dec 21 '22 at 10:25