python selenium - can't click consent

Question

I'm trying to scrape a specific server that requests consent right after connecting. Once I click "I agree" manually, all works fine. But I would like to automate the entire process, including the initial consent click. For that, I need to locate the button to click with selenium - but I can't find the corresponding element in the page source. Any help would be very much appreciated.

Code:

from selenium import webdriver

url = 'https://www.sreality.cz/'
driver = webdriver.Firefox()
driver.get(url)

print(driver.page_source)

Output:

<html><head>
    <meta http-equiv="Content-Type" content="text/html; charset=utf-8">
    <meta name="viewport" content="width=device-width, initial-scale=1.0">
    <title>Seznam.cz Consent</title>
    <script src="//h.seznam.cz/js/dot-small.js"></script>
    <script src="//h.seznam.cz/js/cmp2/scmp-consent.js"></script>
    <style>
      body {
        height: 100%;
        margin: 0;
        overflow: hidden;
      }

      div.scmp.scmp_Dialog-wrapper {
        background: none !important;
      }

      div.scmp_dialog-overlay {
        background-color: unset;
      }

      iframe#bg-iframe {
        height: 100vh;
        width: 100vw;
        pointer-events: none;
        -webkit-filter: blur(5px);
        filter: blur(5px);
      }
    </style>
  <style id="scmp-noscroll">.scmp-noscroll { height: 100vh; height: -webkit-fill-available; margin: 0; overflow: hidden; }</style></head>
  <body class="scmp-noscroll">
    <script>
      var backURLName = '$backUrl';

      function getCookie (cName) {
        var name = cName + '=';
        var cookies = decodeURIComponent(document.cookie);
        var allCookies = cookies.split(';');
        var len = allCookies.length;

        for (var i = 0; i < len; i++) {
          var c = allCookies[i];
          while (c.charAt(0) === ' ') {
            c = c.substring(1);
          }
          if (c.indexOf(name) === 0) {
            return c.substring(name.length);
          }
        }
        return null;
      }

      var backURL = getCookie(backURLName);

      if (backURL) {
        var iframe = document.createElement('iframe');

        iframe.setAttribute('src', backURL);
        iframe.setAttribute('id', 'bg-iframe');
        iframe.setAttribute('sandbox', '');
        iframe.setAttribute('frameborder', '0');

        document.body.appendChild(iframe);
      }
    </script>
  

<iframe src="https://h.seznam.cz/html/cmp.html?v=2.292.0" style="border: 0px none; display: none;"></iframe><div class="szn-cmp-dialog-container" style="margin: unset; padding: unset; width: unset; height: 100%;"></div></body></html>

In browser:

Its a duplicate question. Check this answer for solution - https://stackoverflow.com/a/76053412/7598774 — Shawn, Apr 24 '23 at 08:10
@JeffC - The link to the answer you have shared has `#shadow-root(open)`. In this case it is `shadow-root(closed)`. I don't think the solution to handle this using `.shadow_root` method is correct. — Shawn, Apr 24 '23 at 14:37

score 0 · Answer 1 · answered Apr 24 '23 at 08:14

0

Looking at the page, the consent popup seems to be a shadow DOM element, hidden within the last div.

Here is a related question: How to automate shadow DOM elements using selenium?

answered Apr 24 '23 at 08:14

ricky

21
5

Thank you for the information. I can't make it work, it's not clear to me how to reference that last div (there is no ID). Any chance you could elaborate your answer? – polTo Apr 24 '23 at 11:35
(By.XPATH, "//div[@class='szn-cmp-dialog-container']") should identify the div as long as the class attribute is unique to that div. – ricky Apr 24 '23 at 12:49

python selenium - can't click consent

1 Answers1