0

Super Noob question. I'm getting ready for my fantasy football draft and rather than copy/paste rankings from websites I wanted to script it out to save some time in the future. I'm looking at yahoo's rankings and there is one table that changes when you select the different positions ("QB", "RB", "WR", etc.) in a selector above the table. The url doesn't change but my script is only giving me the first table ("QB") data. How do I write this to include the other "hidden" tables?

import re
from urllib.request import urlopen
from bs4 import BeautifulSoup

url = "https://sports.yahoo.com/nfl/fantasy-rankings/"
html = urlopen(url)
soup = BeautifulSoup(html, 'lxml')

rows = soup.find_all('tr')

list_rows = []
for row in rows:
    cells = row.find_all('td')
    str_cells = str(cells)
    clean = re.compile('<.*?>')
    clean2 = (re.sub(clean, '',str_cells))
    list_rows.append(clean2)

df = pd.DataFrame(list_rows)
df1 = df[0].str.split(',', expand=True)
Brian
  • 1
  • 2
    The page content is dynamically generated. You cannot get it with BS. Should use Selenium. – DYZ Aug 17 '18 at 17:25
  • As suggested, Selenium. https://stackoverflow.com/questions/45867355/beautiful-soup-fetch-dynamic-table-data – tgikal Aug 17 '18 at 17:43

0 Answers0