BeautifulSoup returning 'None' when element definitely exists

Question

firstly-I apologize if I'm missing something super simple, I've looked at many questions and cannot find this out for the life of me.

Basically, the website I'm trying to gather text is here:

https://www.otcmarkets.com/stock/MNGG/overview

I want to pull the information from the side that says 'Dark or Defunct,' my current code is as follows:

url = 'https://www.otcmarkets.com/stock/MNGG/overview'
page = requests.get(url)
soup = BeautifulSoup(page.content, "html.parser")
ticker = soup.find('href', 'Dark or Defunct')

But as the title says, it just returns none. Where am I going wrong? I'm quite inexperienced so I'd love an explanation if possible.

try `ticker = soup.find(text='Dark or Defunct')`, I think that should work — Balduin, Feb 26 '21 at 17:16
it may be dynamically loaded. Search some SO posts regarding selenium and beautifulsoup. i can't look at the moment, but I think that's your issue with this site. — Jonathan Leon, Feb 26 '21 at 17:58

score 0 · Answer 1 · answered Feb 26 '21 at 18:24

It's returning none because it there is no mention of it in the HTML page source. Everything on that website is dynamically loaded from JavaScript sources. BeautifulSoup is designed to pull data out of HTML and XML files, and in the HTML file provided, there is no mention of "Dark or Darker" (so BeautifulSoup correctly finds nothing). You'll need to use a scraping method that has support for JavaScript. See Web-scraping JavaScript page with Python.

BeautifulSoup returning 'None' when element definitely exists

1 Answers1