0

I am working on a project scraping a table off a web site. I am not able to give full code as this is a company specific site with a login. Basically, I am trying to fetch the records from webtable where nore than 100 records present but at a time only 12 records are displaying on screen.I tried to get records using scrollbar but it failed to do so.Kindly help me to get all records in excel sheet.Thanks in advance. I am adding my code below.

from lxml import etree
import urllib.request
import requests
import pandas as pd
from bs4 import BeautifulSoup
url = "https//"
pd.read_html(requests.get(url,verify=False))
df = pd.read_html(url)

I am getting below error:

TypeError: cannot parse from 'Response'

dipesh rane
  • 1

1 Answers1

0

You need to change

pd.read_html(requests.get(url,verify=False))

to

pd.read_html(requests.get(url,verify=False).content)
Roman
  • 1,883
  • 2
  • 14
  • 26
  • Hi, Thanks for your help!! We tried the solution you have been already provided but after using pd.read_html(requests.get(url,verify=False).content) It's giving error as "No tables found" Could you please guide us futher. – dipesh rane Jul 05 '20 at 08:24
  • Are you sure that `````` tag is present on your page? also, check this answer - https://stackoverflow.com/questions/53398785/pandas-read-html-valueerror-no-tables-found
     – Roman Jul 05 '20 at 12:13