BeautifulSoup cannot find data from page

Question

I am trying to pull data from my NFL Fantasy football league using the following code

site = "http://fantasy.nfl.com/league/XXXXXXX"
req = requests.get(site)
soup = BeautifulSoup(req.text,"html5lib")
all_tables=soup.find_all('table')
all_tables

But this returns nothing. I looked into what the variable soup returns, and did control-f to search for player names or stats but had no luck. I have looked at other threads, read some documentation, and tried different parsers like "html.parser", "lxml", and "xml". But still can't figure it out.

Could it be that it is impossible to view this data through python?

score 2 · Answer 1 · answered Sep 17 '18 at 18:14

2

So most likely the table is loaded through javascript. So beautifulSoup won't pickup the table. What I've done in the past is use selenium and load the page and then you can pick up the tables loaded to javascript. I believe there are some libraries now that combine JavaScript and BeautifulSoup.

Maybe take a look at this: Parse the JavaScript returned from BeautifulSoup

answered Sep 17 '18 at 18:14

seriousammy

46
1

I went the Selenium route and it seems to be working so far. Thanks for the help! – Jordan Sep 18 '18 at 00:27
@Jordan I'm glad it worked out! – seriousammy Sep 18 '18 at 20:16

BeautifulSoup cannot find data from page

1 Answers1