0

This is the code that wrote to fetch data from webpage:

import urllib.request import urllib from bs4 import BeautifulSoup

def make_soup(url):
    req=urllib.request.Request(url,headers={'User-Agent': 'Mozilla/5.0'})
    thepage=urllib.request.urlopen(req)
    soupdata=BeautifulSoup(thepage,'html5lib')
    return soupdata

soup=make_soup("https://www.nseindia.com/live_market/dynaContent/live_analysis/top_gainers_losers.htm?cat=G")
t=soup.findAll('table')[0]


for record in t.findAll('tr'):
    print(record.td.text)

'''

for record in t.findAll('tr'):
    for data in record.findAll('td'):
        print(data.text)
'''

But this code fetches only first tr. how to get values for remaining tr

DisappointedByUnaccountableMod
  • 6,656
  • 4
  • 18
  • 22
rajat garg
  • 53
  • 1
  • 1
  • 5
  • 2
    The site uses js to write tds . You 'll have to use selenium – t.m.adam May 12 '17 at 09:42
  • Thanks @t.m.adam for the ans.if you please can tell me how to do this using selenium that would be a great help to me. – rajat garg May 12 '17 at 12:05
  • 1
    Read this post: [parse-a-website-using-selenium-and-beautifulsoup](http://stackoverflow.com/questions/13960326/how-can-i-parse-a-website-using-selenium-and-beautifulsoup-in-python?answertab=votes#tab-top) – t.m.adam May 12 '17 at 12:10
  • @t.m.adam thanks it works like a charm. but as we use selenium it opens a browser window every time i run the script.but i want to run the script in every five minutes and capture the data without opening the browser everytime. can we also do that? – rajat garg May 12 '17 at 12:46
  • I'm not very familiar with selenium . Google it ? – t.m.adam May 12 '17 at 12:51

0 Answers0