0

I am using selenium python to scrape a website and below is my close

from selenium import webdriver

driver = webdriver.Chrome()
driver.get("https://www.businesstimes.com.sg")

whenever I run this code I see the below error enter image description here

It works fine for "https://www.reutersagency.com" but not working for the above. How can I access that and solve this error?

Thanks.

Azeem112
  • 337
  • 1
  • 8
  • 23
  • Either their website has technical difficulties as they explain, or you hit a kind of scraping protection. – Klaus D. Jul 05 '23 at 07:08

1 Answers1

0

I can get the https://www.businesstimes.com.sg/ website normally

you should try those codes from
Selenium webdriver: Modifying navigator.webdriver flag to prevent selenium detection

def edgeopen(driverpath, pdfpath):
    service=Service(executable_path=driverpath)
    edge_options = EdgeOptions()
    #https://stackoverflow.com/questions/53039551/selenium-webdriver-modifying-navigator-webdriver-flag-to-prevent-selenium-detec
    edge_options.add_experimental_option('excludeSwitches', ['enable-automation'])
    edge_options.add_experimental_option('useAutomationExtension', False)
    edge_options.add_argument('lang=zh-CN,zh,zh-TW,en-US,en')
    edge_options.add_argument('--disable-gpu')
    edge_options.add_argument("disable-blink-features=AutomationControlled")#就是这一行告诉chrome去掉了webdriver痕迹
    
    driver = webdriver.Edge(options=edge_options, service = service)
    driver.execute_script("Object.defineProperty(navigator, 'webdriver', {get: () => undefined})")
    driver.execute_cdp_cmd('Network.setUserAgentOverride', {"userAgent": 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/83.0.4103.53 Safari/537.36'})
    driver.set_script_timeout(130)
    
    return driver
Jiu_Zou
  • 463
  • 1
  • 4