1

I am trying to scrape ~10k different urls, using Scrapy with Selenium. At some websites, all I can scrape is the cookie pop-up. How to handle those pop-ups in general? Can the selenium driver accept all cookies by default, so that the cookie pop-up would not render? Navigating the pop-ups with selectors seems impossible, as they differ from page to page.

undetected Selenium
  • 183,867
  • 41
  • 278
  • 352
braythe
  • 13
  • 3
  • 1
    there is NO one, universal method to show pop-ups - so there is NO one, universal method to close them. – furas Jun 07 '22 at 22:03

1 Answers1

1

Different websites are built using different components following different architecture. Some websites may use Vue.js where as some other may use Ember.js.

So in these cases the effective locator strategy to locate the Accept All button on different website is ideally bound to be different.

Hence there can be no generic solution to accept all cookies by default across all websites.

undetected Selenium
  • 183,867
  • 41
  • 278
  • 352