0

Currently, I'm attempting to scrape a few retail websites to compare prices using Selenium and BeautifulSoup. However, there's a particular page Selenium won't load. The browser opens, but the page remains blank. I've tried googling to check if there are certain websites Selenium has difficulty accessing, but turned up nothing. I was using the Gecko driver for Firefox, and changed to the Chromedriver to make sure there wasn't an issue with the particular browser.

Code:

URL = "https://shop.coles.com.au/a/national/everything/browse/pantry/breakfast?pageNumber=1"

browser = webdriver.Chrome(executable_path="C:\ChromeDriver\chromedriver")
browser.get(URL)
Bl3akMirai
  • 25
  • 6
  • What information are you trying to get from the page? – Andrej Kesely Aug 09 '20 at 10:39
  • @AndrejKesely just prices and names of items. – Bl3akMirai Aug 09 '20 at 10:53
  • 1
    Looks like it's detecting selenium via a fingerprint. This may have some answers https://stackoverflow.com/questions/33225947/can-a-website-detect-when-you-are-using-selenium-with-chromedriver. Unfortunately there isn't an API end point to grab the data unfortunately! I tried with headless browser as sometimes it can go around it, but unfortunately it wasn't able to bypass. – AaronS Aug 09 '20 at 12:18
  • @AaronS Didn't realize that was a thing. Really frustrating. Thanks for the link. – Bl3akMirai Aug 09 '20 at 12:26

0 Answers0