Why does the website block the browser navigation when using Selenium?

Question

I read this topic, but it doesn't helped me.

I'm trying to get some element, while it is my first action with the web.

Also, sometimes, on my first connection to the website, the request become broken and the website refuse anything

How does the website know that I'm using selenium? I didn't do any pattern action or fast actions..

This is the website

Can anybody help me to figure this problem?

Here is my code

import selenium
from selenium import webdriver


def open_browser():
    chrome_options = webdriver.ChromeOptions()
    chrome_options.add_argument("--disable-infobars")
    browser = selenium.webdriver.Chrome("./chromedriver2.46.exe", chrome_options=chrome_options)
    browser.maximize_window()
    return browser


def go_to_yad2(browser):
    browser.get("https://www.yad2.co.il/products/all")


def open_category(browser):
    options_object = browser.find_element_by_xpath("//ul[@data-name='salesCatID']")
    print(options_object.text())


def main_method():
    browser = open_browser()
    go_to_yad2(browser)
    open_category(browser)


main_method()

_sometimes, on my first connection to the website_ First meaning what? First of the day? First of the hour? First of the minute? — John Gordon, Mar 16 '19 at 15:07
First of the minute. I mean, changed my ip on vpn, trying to connect via selenium, then blocked. — Some Guy, Mar 16 '19 at 15:08
Either you're doing the vpn wrong and your ip address doesn't actually change, or the website is detecting you based on something other than your ip address. — John Gordon, Mar 16 '19 at 15:18
Nah, I see the website reloading, but after few seconds its again getting blocked — Some Guy, Mar 16 '19 at 15:24
YMMV: https://www.scrapehero.com/how-to-prevent-getting-blacklisted-while-scraping/ — orde, Mar 16 '19 at 18:24

undetected Selenium · Answer 1 · 2019-03-16T20:15:38.717

It is a bit unclear from your question exactly with which element were you trying to interact with and why on your first connection to the website, the request become broken and the website refuse anything.

Some more information interms of message from the website, exception would have helped us to debug the issue in a better way. However, I took your own code and did a couple of simple modifications and was able to extract text as follows:

Code Block:

from selenium import webdriver
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC

def open_browser():
    chrome_options = webdriver.ChromeOptions()
    chrome_options.add_argument('start-maximized')
    chrome_options.add_argument('disable-infobars')
    chrome_options.add_argument('--disable-extensions')
    browser = webdriver.Chrome(chrome_options=chrome_options, executable_path=r'C:\WebDrivers\chromedriver.exe')
    return browser

def go_to_yad2(browser):
    browser.get("https://www.yad2.co.il/products/all")

def open_category(browser):
    print(WebDriverWait(browser, 30).until(EC.visibility_of_element_located((By.XPATH, "//h3[@class='search_title']"))).text)

def main_method():
    browser = open_browser()
    go_to_yad2(browser)
    open_category(browser)

main_method()

Console Output (non-english characters doesn't renders on my localhost):

Why does the website block the browser navigation when using Selenium?

1 Answers1