Failed to obtain response using Selenium python to automate visiting a website

Question

Basically what I want to do is to automate visiting this website. If you visit using Chrome, manually picking report type and date, a download link will appear. I tried to automate this process using python + selenium by following the suggestions in the linked question. But selenium clicking the Get Data button didn't work. I don't know why. Please help.

Please see this link https://stackoverflow.com/a/68598528/12469120 for more context about how to use DatePicker.

Her is my code.

from selenium import webdriver
from selenium.webdriver.support.ui import Select
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.common.keys import Keys


import time
driver = webdriver.Chrome()
driver.execute_cdp_cmd("Page.addScriptToEvaluateOnNewDocument", {
    "source":
        "const newProto = navigator.__proto__;"
        "delete newProto.webdriver;"
        "navigator.__proto__ = newProto;"
    })
driver.get("https://www.nseindia.com/products/content/derivatives/equities/archieve_fo.htm")

time.sleep(5)
datepicker = driver.find_element_by_id("date")
datepicker.click()

selectMonth = driver.find_element_by_xpath('//select[@class="ui-datepicker-month"]')
for option in selectMonth.find_elements_by_tag_name('option'):
    if option.text == 'Mar':
        option.click() 
        break

selectYear = driver.find_element_by_xpath('//select[@class="ui-datepicker-year"]')
for option in selectYear.find_elements_by_tag_name('option'):
    if option.text == '2018':
        option.click() 
        break 

days = driver.find_elements_by_xpath('//a[@class="ui-state-default"]')
days[4].click()
time.sleep(2)
select = Select(driver.find_element_by_id('h_filetype'))
select.select_by_visible_text('Market Activity Report')

wait = WebDriverWait(driver, 10)
element = wait.until(EC.element_to_be_clickable((By.CLASS_NAME, 'getdata-button')))
element.click()
time.sleep(20)

Update

The site can detect Selenium driven browsers. A workaround is added in the code. The download link can be obtained.

Answer 1

The issue is that, there is no file available for the specified date.

No file found for specified date. Try another date.

Though, there were some tweak needed, I have made some changes, see below :-

driver = webdriver.Chrome()
driver.maximize_window()
driver.implicitly_wait(30)
driver.get("https://www.nseindia.com/products/content/derivatives/equities/archieve_fo.htm")
wait = WebDriverWait(driver, 20)
wait.until(EC.element_to_be_clickable((By.CSS_SELECTOR, "input[id='date']"))).click()

select_month = Select(wait.until(EC.visibility_of_element_located((By.XPATH, "//select[@class='ui-datepicker-month']"))))
select_month.select_by_visible_text('Mar')

select_year = Select(wait.until(EC.visibility_of_element_located((By.XPATH, "//select[@class='ui-datepicker-year']"))))
select_year.select_by_visible_text('2017')

days = driver.find_elements_by_xpath("//a[@class='ui-state-default']")
days[3].click()
time.sleep(2)

select = Select(wait.until(EC.element_to_be_clickable((By.ID, "h_filetype"))))
select.select_by_value('fomkt')

element = wait.until(EC.element_to_be_clickable((By.CLASS_NAME, 'getdata-button')))
element.click()

Update 1:

look, this css selector represent the 'file not available' message

div#spanDisplayBox td

it's actually working manually, but hangs with automation (chrome browser). I have faced this issue earlier as well with the same site. However, I have added a method that could help us in following way :

element = wait.until(EC.element_to_be_clickable((By.CLASS_NAME, 'getdata-button')))
element.click()

try:
    if(len(driver.find_elements(By.CSS_SELECTOR, "div#spanDisplayBox td"))) > 0:
        print("Response is present")
        print(wait.until(EC.element_to_be_clickable((By.CSS_SELECTOR, "div#spanDisplayBox td"))).text)
    else:
        print("There is no response")
except:
    print("something went wrong")
    pass

However, when I ran this I got this output :-

There is no response

I would recommend you to check this with Firefox, (cross browser, cross platform and see if that helps).