1

I have been trying to webscrape some information from a webside with Selenium in Python. I load the webside, but I can't accept the cookies, which is necessary to continue. I have tried to use explicitly wait as I suspected that the problem was that the webside loaded before the "accept" bottom becomes clickable.

My code is:

import os
import until as until
from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC

os.environ['PATH'] += r"C:\Users\BackUp HDL\AppData\Local\Programs"
driver = webdriver.Chrome()
driver.get("https://watchmedier.dk/latest/filtered? sitesFilter=policywatch.dk&sitesFilter=shippingwatch.dk&sitesFilter=mobilitywatch.dk&sitesFilter=energiwatch.dk&sitesFilter=finanswatch.dk&sitesFilter=ejendomswatch.dk&sitesFilter=mediawatch.dk&sitesFilter=agriwatch.dk&sitesFilter=fodevarewatch.dk&sitesFilter=medwatch.dk&sitesFilter=kapwatch.dk&sitesFilter=itwatch.dk&sitesFilter=ctwatch.dk&sitesFilter=watchmedier.dk&sitesFilter=advokatwatch.dk")

WebDriverWait(driver, 10).until(EC.element_to_be_clickable((By.XPATH, '//*[@id="notice"]/div[3]/button[2]')))
driver.find_element_by_xpath('//*[@id="notice"]/div[3]/button[2]').click()

Whenever I try to run the code I get an error. Can someone please help me out?

undetected Selenium
  • 183,867
  • 41
  • 278
  • 352
Sofie
  • 17
  • 4

1 Answers1

0

The element Accepter is within an <iframe> so you have to:

  • Induce WebDriverWait for the desired frame to be available and switch to it.

  • Induce WebDriverWait for the desired element to be clickable.

  • You can use either of the following Locator Strategies:

    • Using CSS_SELECTOR:

      driver.get("https://watchmedier.dk/latest/filtered? sitesFilter=policywatch.dk&sitesFilter=shippingwatch.dk&sitesFilter=mobilitywatch.dk&sitesFilter=energiwatch.dk&sitesFilter=finanswatch.dk&sitesFilter=ejendomswatch.dk&sitesFilter=mediawatch.dk&sitesFilter=agriwatch.dk&sitesFilter=fodevarewatch.dk&sitesFilter=medwatch.dk&sitesFilter=kapwatch.dk&sitesFilter=itwatch.dk&sitesFilter=ctwatch.dk&sitesFilter=watchmedier.dk&sitesFilter=advokatwatch.dk")
WebDriverWait(driver, 20).until(EC.frame_to_be_available_and_switch_to_it((By.CSS_SELECTOR,"iframe[title='SP Consent Message']")))
WebDriverWait(driver, 20).until(EC.element_to_be_clickable((By.CSS_SELECTOR, "button[title='Accepter']"))).click()

    • Using XPATH:

      driver.get("https://watchmedier.dk/latest/filtered? sitesFilter=policywatch.dk&sitesFilter=shippingwatch.dk&sitesFilter=mobilitywatch.dk&sitesFilter=energiwatch.dk&sitesFilter=finanswatch.dk&sitesFilter=ejendomswatch.dk&sitesFilter=mediawatch.dk&sitesFilter=agriwatch.dk&sitesFilter=fodevarewatch.dk&sitesFilter=medwatch.dk&sitesFilter=kapwatch.dk&sitesFilter=itwatch.dk&sitesFilter=ctwatch.dk&sitesFilter=watchmedier.dk&sitesFilter=advokatwatch.dk")
WebDriverWait(driver, 20).until(EC.frame_to_be_available_and_switch_to_it((By.XPATH,"//iframe[@title='SP Consent Message']")))
WebDriverWait(driver, 20).until(EC.element_to_be_clickable((By.XPATH, "//button[@title='Accepter']"))).click()

  • Note : You have to add the following imports :

     from selenium.webdriver.support.ui import WebDriverWait
 from selenium.webdriver.common.by import By
 from selenium.webdriver.support import expected_conditions as EC

  • Browser Snapshot:

Watchmedier

Reference

You can find a couple of relevant discussions in:

undetected Selenium
  • 183,867
  • 41
  • 278
  • 352
  • Thanks a lot for the help. The code worked. Now I'm just stuck at a new problem as I'm totally new at using both Python and Selenium. I guess, my skill level simply isn't great enough to extract data in this way. Do you now if there is someone I can contact for more help, since I need the data for my master? – Sofie Dec 04 '21 at 06:56
  • Of coarse, feel free to discuss with me at length in [Selenium](https://chat.stackoverflow.com/rooms/223360/selenium) room. – undetected Selenium Dec 04 '21 at 07:15