I am quite new to using beauifulsoup, I am trying to scrape the text from the website using the code below. However, the find_all returns nothing.
import bs4 as bs
import urllib.request
source = urllib.request.urlopen('https://beta.regulations.gov/document/USCIS-2019-0010-9175').read()
soup = BeautifulSoup(page.content,'html.parser')
text = soup.find_all(class_="px-2")
print(text)
As stated in the comments, the data is loaded dynamically via Javascript. But when you open Firefox/Chrome network tab, you can see from where the data is coming from:
import requests
url = 'https://beta.regulations.gov/document/USCIS-2019-0010-9175'
ajax_url = 'https://beta.regulations.gov/api/documentdetails/{}'
document_id = url.split('/')[-1]
data = requests.get(ajax_url.format(document_id)).json()
# from pprint import pprint # <-- uncoment to see all data
# pprint(data)
print(data['data']['attributes']['content'])
Prints:
Rescind the increase in fees. This is draconian. For all intents and purposes, denying access to this information will prevent many Americans from knowing where they came from. This is an outrage. This is not the mark of a democracy. I strongly disagree with this fee increase