1

I am using Selenium to open different pages of a site. Have tried multiple times but the browser does not open a second webpage after the initial GET call. Have tried on both Chrome and Safari. Here is my code:

driver = webdriver.Chrome()
driver.get("https://www.justdial.com/Chennai/Hr-Consultancy-Services/nct-10258625/page-2")
driver.set_page_load_timeout(30)
driver.get("https://www.justdial.com/Chennai/Hr-Consultancy-Services/nct-10258625/page-3")

Here is the error I get for the second call:

enter image description here

The info from Network logs is Error 504, but I have verified that it works perfectly when done on another window of the browser, without automation

Ajay Kumar
  • 1,253
  • 1
  • 11
  • 17
  • It looks like the page DOES open it's just opened with an errror from the webserver. – DMart Feb 01 '21 at 15:20
  • Yes, it is a server error, but no more clues to help resolve the same. The same is fine for some other 2 sites – Ajay Kumar Feb 01 '21 at 15:28
  • I wonder if there is more to the URL action than a simple GET is providing. Are you sure that is the correct and full URL? Can you execute this manually? What does the network tab in the browser say is happening when run manually? – DMart Feb 01 '21 at 15:30
  • The code worked perfectly fine for me. I used Chrome too. – Ananth Feb 01 '21 at 16:36

2 Answers2

3
options.add_experimental_option(
    "excludeSwitches", ['enable-automation'])

options.add_argument(
    "user-agent=Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/87.0.4280.88 Safari/537.36")
options.add_argument("--remote-debugging-port=9222")

driver = webdriver.Chrome(options=options)
driver.get(
    "https://www.justdial.com/Chennai/Hr-Consultancy-Services/nct-10258625/page-2")
driver.set_page_load_timeout(30)
driver.get(
    "https://www.justdial.com/Chennai/Hr-Consultancy-Services/nct-10258625/page-3")

the website is detecting automation use above code :)

You can also do this in single line

Just add below argument:

options.add_argument('--disable-blink-features=AutomationControlled')

disabling enable-automation , or disabling automation controller disables webdriver.navigator which few websites uses to detect automation scripts

PDHide
  • 18,113
  • 2
  • 31
  • 46
2

A bit of more information about your usecase would have helped to construct a more canonical answer. However I was able to access the Page 2 of justdial.com/Chennai/Hr-Consultancy-Services with a minimized code block as follows:

  • Code Block:

    from selenium import webdriver

options = webdriver.ChromeOptions() 
options.add_argument("start-maximized")
driver = webdriver.Chrome(options=options, executable_path=r'C:\WebDrivers\chromedriver.exe')
driver.get("https://www.justdial.com/Chennai/Hr-Consultancy-Services/nct-10258625/page-2")

  • Browser Snapshot:

justdial

But while sending multiple get() one after another:

driver.get("https://www.justdial.com/Chennai/Hr-Consultancy-Services/nct-10258625/page-2")
driver.get("https://www.justdial.com/Chennai/Hr-Consultancy-Services/nct-10258625/page-3")

It seems ChromeDriver initiated Chrome Browser gets detected and the following error is shown:

An error occurred while processing your request.
Reference #97.e5732c31.1612205693.6fd2708

Solution

To avoid the detection you can add the following option:

--disable-blink-features=AutomationControlled

Example

from selenium import webdriver

options = webdriver.ChromeOptions() 
options.add_argument("start-maximized")
options.add_argument('--disable-blink-features=AutomationControlled')
driver = webdriver.Chrome(options=options, executable_path=r'C:\WebDrivers\chromedriver.exe')
driver.get("https://www.justdial.com/Chennai/Hr-Consultancy-Services/nct-10258625/page-2")
driver.get("https://www.justdial.com/Chennai/Hr-Consultancy-Services/nct-10258625/page-3")
undetected Selenium
  • 183,867
  • 41
  • 278
  • 352
  • it seems that nowadays (2023) use more complex detection and `start-maximized` and `--disabled-blink-features=AutomationControlled` alone it's not enough. – RubenLaguna Jun 15 '23 at 08:26