BeautifulSoup find.text returns empty string although element exists

Question

I am scraping the following; https://www.espn.com/nfl/scoreboard/ and trying to get the times of the games

import requests
from bs4 import BeautifulSoup

r = requests.get("https://www.espn.com/nfl/scoreboard")
soup = BeautifulSoup(r.content, "lxml")

for section in soup.find_all("section", {"class": "Card gameModules"}):
    for game in section.find_all("section", {"class": "Scoreboard bg-clr-white flex flex-auto justify-between"}):
        print(game.find("div", {"class": "ScoreCell__Time ScoreboardScoreCell__Time h9 clr-gray-03"}).text)

Even though it should return the times of the games, it just returns empty strings. Why is this?

That content is most likely to be dynamically loaded with javascript. In that case `reuqests` can't see it. Test it by reviewing the output of `print(r.convent)` or by accessing the page with javascript disabled in the browser. — DeepSpace, Dec 21 '22 at 18:10
Go bring up that web page. Now do Ctrl-U to "View Page Source". THAT'S what `requests` sees. It has templates to fill in, but none of the data. The data is supplied later via Javascript. — Tim Roberts, Dec 21 '22 at 18:20
Are you sure the `div` you're looking for exists ? Can you save the div you are calling a `find` on and paste the output of `div.prettify()` — lonetwin, Dec 21 '22 at 18:21
[Canonical](https://stackoverflow.com/questions/8049520/web-scraping-javascript-page-with-python) — ggorlen, Dec 21 '22 at 18:53

Md. Fazlul Hoque · Answer 1 · 2022-12-21T23:10:47.240

Your html class element selection ScoreCell__Time ScoreboardScoreCell__Time h9 clr-gray-03 is correct but empty that's why you are getting empty ResultSet. Select the right content related to the element, then you will get output:

Example:

import requests
from bs4 import BeautifulSoup

r = requests.get("https://www.espn.com/nfl/scoreboard")
soup = BeautifulSoup(r.content, "lxml")

for section in soup.find_all("section", {"class": "Card gameModules"}):
    for game in section.find_all("section", {"class": "Scoreboard bg-clr-white flex flex-auto justify-between"}):
        print(game.find("div", {"class": "ScoreCell__TeamName ScoreCell__TeamName--shortDisplayName truncate db"}).text)

Output:

Jaguars
Bills
Saints
Texans
Seahawks
Giants
Bengals
Lions
Falcons
Commanders
Eagles
Raiders
Packers
Broncos
Buccaneers
Chargers

BeautifulSoup find.text returns empty string although element exists

1 Answers1