As it turns out, there's nothing wrong with your CSS selector (either the original version you show, or the modified version used in your code; both select the same content). The problem is that the portion of the page you want is dynamically generated by JavaScript, so it's not in the HTML. To see this, try:
from bs4 import BeautifulSoup
import requests
html = requests.get(
'https://www.cryptocompare.com/coins/bnb/influence/USDT'
).text
with open('output.txt', 'w') as f:
f.write(html)
If you search the resulting file for "We don't have any code repository data yet."
, you'll see that it's not there.
So we need to not only access the HTML page, but also run its JavaScript, and save the resulting page's source. You can do this with Selenium, as explained in this answer. You will first have to install Selenium (pip install selenium
should probably work), and download the geckodriver executable as appropriate for your OS.
With that done, your code (with Mr.Manhattan's addition) would look like this:
from contextlib import closing
from bs4 import BeautifulSoup
from selenium.webdriver import Firefox
with closing(Firefox()) as browser:
browser.get('https://www.cryptocompare.com/coins/bnb/influence/USDT')
html = browser.page_source
soup = BeautifulSoup(html, 'html.parser')
total_commit_node = soup.select_one(
'#col-body > div > social-influence > '
'div.row.row-zero.influence-others.panel-inactive > div:nth-child(3) > h4'
)
if total_commit_node:
print(total_commit_node.text)
else:
print('Could not match CSS selector')
(Note: I split the selector onto two lines to avoid scrolling. It functions exactly the same, since consecutive string literals are automatically concatenated.)
For me, this correctly outputs:
We don't have any code repository data yet.