1

Using python, I am trying to get the number in front of the "Advances -" text on the url: https://www.nseindia.com/market-data/pre-open-market-cm-and-emerge-market

I am using the below xpath:

//*[@id="livepreOpenAdv"]

but getting the below error:

no such element: Unable to locate element: {"method":"xpath","selector":"//*[@id="livepreOpenAdv"]"} (Session info: headless chrome=83.0.4103.61)

I am using chromedriver in Linux (ubuntu 20.04)

Below is the python code:

from selenium import webdriver
from selenium.webdriver.chrome.options import Options

strURL="https://www.nseindia.com/market-data/pre-open-market-cm-and-emerge-market"

options = Options()

options.add_argument('--headless')
options.add_argument('--no-sandbox')
options.add_argument('--disable-gpu')

driver = webdriver.Chrome("/usr/lib/chromium-browser/chromedriver",options=options)

driver.implicitly_wait(60)
driver.get(strURL)

print("Finding element.")

try:
    # advances = driver.find_element_by_id("livepreOpenAdv")
    advances = driver.find_element_by_xpath('''//*[@id="livepreOpenAdv"]''').text
    print(advances)
except Exception as ex:
    print(ex)

driver.close()

print("closed driver")

This code works fine in windows. Not able to make out what is the issue here.

Rajesh
  • 11
  • 3
  • You need to use `WebDriverWait`. I was able to get 39 sometimes and sometimes I don't get it. This starting point can be helpful. – Jonathan Lam May 28 '20 at 23:13

2 Answers2

0

Add the following imports :

from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC

and replace the line starting with advances = driver... with :

advances = WebDriverWait(driver, 20).until(EC.visibility_of_element_located((By.XPATH, "//span[@id='livepreOpenAdv']"))).getText()
E.Wiest
  • 5,425
  • 2
  • 7
  • 12
  • Getting Timeout exception error: Traceback (most recent call last): File "t.py", line 33, in advances = WebDriverWait(driver,60).until(EC.visibility_of_element_located((By.XPATH, "//span[@id='livepreOpenAdv']"))) File "/home/ubuntu/.local/lib/python3.8/site-packages/selenium/webdriver/support/wait.py", line 80, in until raise TimeoutException(message, screen, stacktrace) selenium.common.exceptions.TimeoutException: Message: – Rajesh May 29 '20 at 08:55
  • Remove the `disable-gpu` option (not required on Linux system) and pass the `no-sandbox` as first argument. Remove also the `driver.implicitly_wait(60)` from your script. Also, be sure to check your Chromedriver version and Chrome browser are compatible. – E.Wiest May 29 '20 at 11:24
  • The chrome browser was not installed in my linux system. I got the chrome installed and with it the chromedriver also got installed. Now both are of same version but still facing the same issue. – Rajesh May 29 '20 at 19:59
  • I tried to print driver.title and it gives me "Access Denied". Trying to find a workaround with that but no luck yet. Other urls like www.blgger.com is working fine. – Rajesh May 29 '20 at 20:23
  • Then you should check : https://stackoverflow.com/a/54433282/9978746 – E.Wiest May 30 '20 at 01:48
0

Finally the below code is working for me. I had to install google-chrome-stable package through apt. Also suggestions from @E.Wiest helped. Thanks! As this was an headless execution, the url was giving access denied message. Added user-agent parameter to resolve that. 

from selenium import webdriver
from selenium.webdriver.chrome.options import Options
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC
import time

strURL="https://www.nseindia.com/market-data/pre-open-market-cm-and-emerge-market"

options = Options()
options.add_argument('--no-sandbox')
options.add_argument('--headless')
options.add_argument(f'user-agent=Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/83.0.4103.61 Safari/537.36')

driver = webdriver.Chrome("/usr/lib/chromium-browser/chromedriver",options=options)

t1=time.time()
driver.get(strURL)

strXpath_advances='//*[@id="livepreOpenAdv"]'
strXpath_decline='//*[@id="livepreOpenDec"]'

try:
    advances = WebDriverWait(driver,30).until(EC.visibility_of_element_located((By.XPATH, strXpath_advances)))
    decline = WebDriverWait(driver,30).until(EC.visibility_of_element_located((By.XPATH, strXpath_decline)))
    print("Advances={} Decline={}".format(advances.text,decline.text))
except Exception as ex:
    print("Exception=",ex)

t2=time.time()-t1
driver.close()
print("Done. closed driver. Time={0:.2f}".format(t2))
Rajesh
  • 11
  • 3