Beautiful Soup fetch dynamic table data

Question

I have the following code:

url = 'https://www.basketball-reference.com/leagues/NBA_2017_standings.html#all_expanded_standings'
html = urlopen(url)
soup = BeautifulSoup(html, 'lxml')

print(len(soup.findAll('table')))
print(soup.findAll('table'))

There are 6 tables on the webpage, but it only returns 4 tables. I tried to use 'html.parser' or 'html5lib' as parsers but did not work either.

Any idea how I can get the Table "expanded standings" from the webpage?

Thanks!

What do you mean? do you know how I can access it? – user2993519 Aug 24 '17 at 17:45 — user2993519, Aug 24 '17 at 17:45
You can access the rest using selenium. – SIM Aug 24 '17 at 19:30 — SIM, Aug 24 '17 at 19:30

MD. Khairul Basar · Answer 1 · 2017-08-26T15:03:50.050

requests can't fetch data that are loaded by JS. So, you have to use selenium. First install selenium via pip - pip install selenium and download chrome driver and put the file in your working directory. Then try the following code.

from bs4 import BeautifulSoup
import time
from selenium import webdriver

url = "https://www.basketball-reference.com/leagues/NBA_2017_standings.html"
browser = webdriver.Chrome()

browser.get(url)
time.sleep(3)
html = browser.page_source
soup = BeautifulSoup(html, "lxml")

print(len(soup.find_all("table")))
print(soup.find("table", {"id": "expanded_standings"}))

browser.close()
browser.quit()

See selenium documentation.

If you are on Linux and get error Chromedriver executable needs to be in the PATH then try following these ways - link-1, link-2

Beautiful Soup fetch dynamic table data

1 Answers1

Linked