3

So I have a webscraper which first of all needs to get past the cookie banner of a given website. Normally I'd just locate the element by id or classname and be done with it, but on this site none of the elements can be located. I've tried/checked the following:

  • The element of interest is <div id="cookiescript_accept" tabindex="0" role="button" data-cs-i18n-text="[]">Alles accepteren</div>
  • The element is not part of an iframe
  • The element is not part of a shadow DOM
  • Using wait.until(ExpectedConditions.visibilityOfElementLocated hits the 15 seconds timeout
  • Using driver.executeScript("return document.getElementById('cookiescript_accept');"); doesn't work either
  • The parent element and parent of parent can also not be found

I'm still fairly new to Selenium and HTML so I must be missing something, please tell me if you know what that is

Code:

public void loadUrl(String url) {
    System.out.println("\t\t- loadUrl " + url);
    idle5000();

    driver.get(url);
    idle5000();

    setWindowSize();
    idle5000();
    printFirefoxCPU();

    scrollViewport();
    idle5000();
    printFirefoxCPU();
}
loadUrl("https://www.schoolplaten.com/");
prepRunnable.getDriver().findElement(By.id("cc-cookiescript_accept")).click();
//  -> NoSuchElementException
WebDriverWait wait = new WebDriverWait(prepRunnable.getDriver(), 15);
wait.until(ExpectedConditions.visibilityOfElementLocated(By.id("cookiescript_accept")));
//  -> Timeout
WebElement elem = (WebElement) prepRunnable.getDriver().executeScript("return document.getElementById('cookiescript_accept');");
elem.click();
//  -> elem is null
Tim U
  • 113
  • 1
  • 6
  • Show use your code. – JonR85 Nov 10 '21 at 01:24
  • has the element rendered on screen by the time your code executes? if not, the you should wait before you find the element, maybe try a classical sleep for 10 seconds and see the results? – tsamridh86 Nov 10 '21 at 04:00

2 Answers2

0

I could locate the web element with the below XPath

//*[name()='div' and @id='cookiescript_accept']

and could perform click on it with the help of below code :

WebDriverWait wait = new WebDriverWait(driver, 30);
wait.until(ExpectedConditions.elementToBeClickable(By.xpath("//*[name()='div' and @id='cookiescript_accept']"))).click();
cruisepandey
  • 28,520
  • 6
  • 20
  • 38
  • For me this code has the same problem as all the other `wait.until` attempts, it hits the timeout and then the element still can't be found – Tim U Nov 10 '21 at 11:53
  • Unbelievable, I used the exact code and it worked like a charm! You have to use this xpath `//*[name()='div' and @id='cookiescript_accept']` – cruisepandey Nov 10 '21 at 11:57
  • `loadUrl("https://www.schoolplaten.com/");` `WebDriverWait wait = new WebDriverWait(driver, 30);` `wait.until(ExpectedConditions.elementToBeClickable(By.xpath("//*[name()='div' and @id='cookiescript_accept']"))).click();` This is literally what I'm running – Tim U Nov 10 '21 at 12:03
  • what error did it throw up? – cruisepandey Nov 10 '21 at 12:04
  • `TimeoutException: Expected condition failed: waiting for element to be clickable: By.xpath: //*[name()='div' and @id='cookiescript_accept'] (tried for 30 second(s) with 500 milliseconds interval)` and right after: `NoSuchElementException: Unable to locate element: //*[name()='div' and @id='cookiescript_accept'` – Tim U Nov 10 '21 at 12:06
  • Well, if you could believe. The code that I have given you works for me well. – cruisepandey Nov 10 '21 at 12:10
0

To click() on the element ALLES ACCEPTEREN you can use either of the following Locator Strategies:

  • cssSelector:

    driver.findElement(By.cssSelector("div#cookiescript_accept")).click();

  • xpath:

    driver.findElement(By.xpath("//div[@id='cookiescript_accept']")).click();

However, the element is a dynamic element so to click() on the element you need to induce WebDriverWait for the elementToBeClickable() and you can use either of the following Locator Strategies:

  • cssSelector:

    new WebDriverWait(driver, 20).until(ExpectedConditions.elementToBeClickable(By.cssSelector("div#cookiescript_accept"))).click();

  • xpath:

    new WebDriverWait(driver, 20).until(ExpectedConditions.elementToBeClickable(By.xpath("//div[@id='cookiescript_accept']"))).click();

Alternative

As an alternative you can also use the executeScript() as follows:

  • cssSelector:

    ((JavascriptExecutor)driver).executeScript("arguments[0].click();", new WebDriverWait(driver, 20).until(ExpectedConditions.elementToBeClickable(By.cssSelector("div#cookiescript_accept"))));

  • xpath:

    ((JavascriptExecutor)driver).executeScript("arguments[0].click();", new WebDriverWait(driver, 20).until(ExpectedConditions.elementToBeClickable(By.xpath("//div[@id='cookiescript_accept']"))));

Reference

You can find a detailed discussion on NoSuchElementException in:

undetected Selenium
  • 183,867
  • 41
  • 278
  • 352