0

enter image description here

''' Trying to fetch Equity Derivatives data from NSE https://www.nseindia.com/->Market Data-> Derivatives Market Works until click action, the browser navigates to Derivatives Market but then thorws access denied error as below

 <h1>
   Access Denied
  </h1>
  You don't have permission to access "http://www.nseindia.com/market-data/equity-derivatives-watch" on this server.
  <p>
   Reference #18.386dcc17.1603823463.54b06d7
  </p>
'''

from selenium import webdriver
from selenium.webdriver import ActionChains
from bs4 import BeautifulSoup
import time

# Tried all possible options below

options = webdriver.ChromeOptions()
options.add_argument("start-maximized")
options.add_argument("disable-infobars")
options.add_argument("--disable-extensions")
options.add_argument('--disable-gpu')
driver = webdriver.Chrome(chrome_options=options)

driver.get("https://www.nseindia.com/")

marketdata = driver.find_element_by_xpath("//*[@id='main_navbar']/ul/li[3]/a")
derivativesmarket = driver.find_element_by_xpath("//*[@id='main_navbar']/ul/li[3]/div/div[1]/div/div[1]/ul/li[3]/a")

actions = ActionChains(driver)
actions.move_to_element(marketdata).move_to_element(derivativesmarket).click().perform()

html = driver.page_source
soup = BeautifulSoup(html, 'html.parser')
# soup = BeautifulSoup(html,'lxml')
time.sleep(7)

print(soup.prettify())[enter image description here][1]
sandeep sangole
  • 1
  • 1

1 Answers1

0

Add

options.add_experimental_option("excludeSwitches", ["enable-automation"])
options.add_experimental_option('useAutomationExtension', False)

and use

 from selenium.webdriver.support.ui import WebDriverWait
 from selenium.webdriver.support import expected_conditions as EC

Then access the element like so:

WebDriverWait(driver, 20).until(EC.visibility_of_element_located((By.CSS_SELECTOR, "h1.h1")))

Reference:- Access Denied You don't have permission to access "site" on this server using ChromeDriver and Chrome through Selenium Python

Abhishek Rai
  • 2,159
  • 3
  • 18
  • 38
  • Thanks for your reply Abrar Ahmed. Its timing out at WebDriverWait method. Getting error as below, line 34, in WebDriverWait(driver, 20).until(EC.visibility_of_element_located((By.CSS_SELECTOR, "h1.h1"))) File "/Users/sandeepsangole/PycharmProjects/nse/venv/lib/python3.8/site-packages/selenium/webdriver/support/wait.py", line 80, in until raise TimeoutException(message, screen, stacktrace) selenium.common.exceptions.TimeoutException: Message: – sandeep sangole Oct 27 '20 at 20:11
  • I am hoping you have changed the `(By.CSS_SELECTOR, "h1.h1")))` to the element you are looking for? – Abhishek Rai Oct 27 '20 at 20:13
  • The `CSS_SELECTOR, "h1.h1")` is from the other answer. There is no such element on YOUR page, hence the error. You need to replace that with the selector of the element you are looking for. – Abhishek Rai Oct 27 '20 at 20:16
  • Changed code as below , driver = webdriver.Chrome(chrome_options=options) driver.get("https://www.nseindia.com/") marketdata = driver.find_element_by_xpath("//*[@id='main_navbar']/ul/li[3]/a") derivativesmarket = driver.find_element_by_xpath("//*[@id='main_navbar']/ul/li[3]/div/div[1]/div/div[1]/ul/li[3]/a") actions = ActionChains(driver) actions.move_to_element(marketdata).move_to_element(derivativesmarket).click().perform() WebDriverWait(driver, 20).until(EC.visibility_of_element_located((By.ID, "tableLiveMarket-equity-derivatives"))) – sandeep sangole Oct 27 '20 at 20:22
  • I'd suggest another look at the `html` of the page you are scraping. Seems more than likely that the element is not being located. I'm off to sleep now. Good luck! – Abhishek Rai Oct 27 '20 at 20:25
  • Thanks for your help Abrar Ahmed. Attached html code image for ref. – sandeep sangole Oct 27 '20 at 20:30
  • I tried everything I know. I think this is not an issue with the code but with the page itself. I tried reading the page with pandas, it is forbidden. So, we are probably back from where we started. The page, it seems is blocking access, hence the failure to locate element. I guess you will try it on your own. – Abhishek Rai Oct 28 '20 at 04:21