4

I am trying to extract table data from this web site

Following is the code--

import requests
from bs4 import BeautifulSoup as bs

page = requests.get('https://www.vitalityservicing.com/serviceapi/Monitoring/QueueDepth?tenantId=1')

soup = bs(page.text, "html.parser")

#None of the following method works
tb = soup.table 
#tb = soup.body.table
#tb = soup.find_all('table')

When I try to print tb its None

So I tried to look at the body of the downloaded HTML with

print(soup.body.prettify())

I dont see the table elements or it's child elements . Only <body> and <script> elements are present:

Output of print(soup.body)

But when I inspect the page in chrome I see all the elements:

table and it's child elements present while inspecting

I don't understand why the table element is not being downloaded with requests.get while its there when I load the page on chrome

spark
  • 1,271
  • 1
  • 12
  • 18
  • Does this answer your question? [Beautiful Soup Can't Find Tags](https://stackoverflow.com/questions/44867425/beautiful-soup-cant-find-tags) – user202729 Jan 18 '21 at 07:10

1 Answers1

6

You are not getting that content because, when you execute the request, it is not present in the page. Yet.

If you check the javascript code between the script tags, you can see it is generating the table dynamically. So, you receive the html code before that has happened, since requests is not a browser and won't execute the js, and you don't get to see the table.

Now that you know why you can't see the table, your next problem is how to get the HTML produced after the javascript execution. Don't faint, it is feasible. You might find the solutions in this question interesting.

Good luck

Pablo M
  • 326
  • 2
  • 7
  • Yup, I figured it out and was using selenium webdriver to load the page. Solved my problem. Thanks – spark Aug 20 '18 at 09:54