Unable to scrape data from Website with Python

Question

I want to extract tables from "Bonds Traded on the Exchange" and "OTC trades"and save it to excel sheet. I am trying to scrape data with python ( BS & requests ) but I am unable to scrape data ( I dont wanna use selenium). Can any1 guide me ? I am not getting any error , it doesn't get prpcessed in python terminal I think terminal gets hanged , as I don't even get any error message .


import requests
import pandas as pd
import os
from bs4 import BeautifulSoup as bs



url = "https://www1.nseindia.com/products/content/debt/corp_bonds/cbm_reporting_homepage.htm"

#condition  True
#while condition:

html = requests.get(url).content
page= requests.get(url)
soup= bs(page.text, 'lxml')
df_list = pd.read_html(html)
df = df_list[0]     # can change 0 to other number 
print(df)

I dont get any user response , website is opening in browser but not fetched via python..selenium works but it is slow..........and what is user agent? — Tech Bro, Sep 09 '21 at 12:20
You're not giving enough information to us in order to help you out. What sort of error message & response code are you getting? It is very important to include such information inside your question. — Bialomazur, Sep 09 '21 at 12:20
I am not getting any error , it doesn't get prcessed in python terminal — Tech Bro, Sep 09 '21 at 12:21
No proxy / No firewall ...can scrape data via selenium but not with BS/Requests/pandas etc...even I am not getting any error ...seems that it gets hanged — Tech Bro, Sep 09 '21 at 12:24
Please don't make more work for others by vandalizing your posts. By posting on the Stack Exchange (SE) network, you've granted a non-revocable right, under a [CC BY-SA license](//creativecommons.org/licenses/by-sa/4.0), for SE to distribute the content (i.e. regardless of your future choices). By SE policy, the non-vandalized version is distributed. Thus, any vandalism will be reverted. Please see: [How does deleting work? …](//meta.stackexchange.com/q/5221). If permitted to delete, there's a "delete" button below the post, on the left, but it's only in browsers, not the mobile app. — Makyen, Sep 17 '21 at 08:06

score 2 · Accepted Answer · answered Sep 09 '21 at 13:11

If you look at Network tab, you will see cbm_reporting_cbricsL.htm which is what you need to scrape. By the way, you should also add headers for requests to work properly. See detailed explanation in this thread:

import requests
import pandas as pd
from bs4 import BeautifulSoup

res = requests.get(
    'https://www1.nseindia.com/products/dynaContent/debt/corp_bonds/htms/cbm_reporting_cbricsL.htm',
    headers={"User-Agent": "Mozilla/5.0 (X11; CrOS x86_64 12871.102.0) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/81.0.4044.141 Safari/537.36"}
    )

soup = BeautifulSoup(res.text, 'lxml')

raw_columns = [row.find_all('td') for row in soup.find_all('tr')]

# first 3 items were dummy
df = pd.DataFrame.from_records(raw_columns[3:])

The result would look like:

0   [INE001A07TA7]  [HOUSING DEVELOPMENT FINANCE CORPORATION LTD S...   [ 100.0030] [ 4.7082]   [ 16]   [[ 168000.00]]  [ 100.0000] [ 4.7091]
1   [INE134E07AP6]  [POWER FINANCE CORPORATION LTD. TRI SRV CATIII...   [ 100.8500] [ 6.6934]   [ 1]    [ 1000.00 ] [ 100.8500] [ 6.6934]
2   [INE020B08963]  [RURAL ELECTRIFICATION CORPORATION LIMITED SR-...   [ 107.6835] [ 5.9200]   [ 1]    [ 1500.00 ] [ 107.6835] [ 5.9200]
3   [INE163N08131]  [-] [ 104.2195] [ 6.6200]   [ 1]    [ 780.00 ]  [ 104.2195] [ 6.6200]
4   [INE540P07343]  [-] [ 104.3408] [ 9.3603]   [ 6]    [[ 1110.00]]    [ 104.2640] [ 9.3800]
... ... ... ... ... ... ... ... ...
93  [INE377Y07250]  [BAJAJ HOUSING FINANCE LIMITED SR 27 5.69 NCD ...   [ 100.0300] [ 5.6845]   [ 1]    [ 9000.00 ] [ 100.0300] [ 5.6845]
94  [INE115A07ML7]  [LIC HOUSING FINANCE LIMITED SRTR349OP-1 7.4NC...   [ 105.0991] [ 5.5000]   [ 1]    [ 1000.00 ] [ 105.0991] [ 5.5000]
95  [INE020B07HN3]  [RURAL ELECTRIFICATION CORPORATION LIMITED SR-...   [ 123.6000] [ 4.4400]   [ 1]    [ 10.00 ]   [ 123.6000] [ 4.4400]
96  [INE101A08070]  [MAHINDRA AND MAHINDRA LIMITED 9.55 NCD 04JL63...   [ 125.5000] [ 7.5218]   [ 1]    [ 820.00 ]  [ 125.5000] [ 7.5218]
97  [INE062A08215]  [STATE BANK OF INDIA SERIES I 8.75 BD PERPETUA...   [ 104.5304] [ 7.0000]   [ 1]    [ 10.00 ]   [ 104.5304] [ 7.0000]

yw, consider [accepting](https://meta.stackexchange.com/questions/5234/how-does-accepting-an-answer-work/5235#5235) my answer — Rustam Garayev, Sep 10 '21 at 07:00
how to accept it ?, I am new here , so Kindly guide me for me — Tech Bro, Sep 10 '21 at 07:16
Check this link: https://meta.stackexchange.com/questions/5234/how-does-accepting-an-answer-work/5235#5235 — Rustam Garayev, Sep 10 '21 at 07:58

Tech Bro · Answer 2 · 2022-07-05T16:20:14.063

-3

Here is what I ended up with. I have used pandas pd to extract 1st table from webpage

import requests
import pandas as pd

headers = {"User-Agent": "Mozilla/5.0 (X11; CrOS x86_64 12871.102.0) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/81.0.4044.141 Safari/537.36"}

html = requests.get(
    'abcd',
    headers=headers).content


df_list = pd.read_html(html)
df = df_list[0]
print (df)

edited Jul 05 '22 at 16:20

answered Sep 10 '21 at 07:59

Tech Bro

23
5

1

Please add further details to expand on your answer, such as working code or documentation citations. – Community Sep 10 '21 at 08:29

Unable to scrape data from Website with Python

2 Answers2