Slow code using selenium and beautifulsoup

Question

I am trying to fetch data using selenium webdriver and beautiful soup from a website. Below segment of code is taking a long time to execute.

time1 = time.time()
options = webdriver.ChromeOptions()
options.add_argument('headless')
driver = webdriver.Chrome(chrome_options=options)
driver.get("https://www.bseindia.com/")
elem = driver.find_element_by_id("suggestBoxEQ")
elem.clear()
elem.send_keys("538707")  
elem.send_keys(Keys.RETURN)
print(driver.current_url)
html = driver.page_source
soup = BeautifulSoup(html, 'lxml')
driver.quit()
time2 = time.time()
print(time2-time1)

It takes 13.876seconds to execute this code. Is there any way to speed up execution time of current code or another faster method to get fetch the data?

Answer 1

After entering 538707 on this URL, the page redirects to this URL.

The tables and other data on this page are loaded from AJAX requests. You can directly get data from these requests for scraping. To see the AJAX requests, go the XHR tab under the Network tab in Developer tools and refresh the page. You can get data from the XHR requests you see here.

For example, the table Securitywise Delivery Position is loaded from this URL. So, you can directly get the table like this:

import requests

r = requests.get('https://www.bseindia.com/stock-share-price/SiteCache/SecurityPosition.aspx?Type=EQ&text=538707')
soup = BeautifulSoup(r.text, 'lxml')
table = soup.find('table')

Scrape this table using BeautifulSoup. For example:

print(table.find('td', class_='newseoscripfig').text)
# 13 Apr 2018

Similarly, you can find nearly all the data that is loaded dynamically in other XHR requests. As Selenium is not used here, the script is pretty fast.