Before using BeautifulSoup on html text of this website, I'm trying to get data using 'requests' library.
But, the output sometimes returns html text and other times it returns 'None'. Can I somehow modify the code to always return html text of the website?
from bs4 import BeautifulSoup
import requests
url = 'https://www.amazon.in/s?k=bags&crid=2M096C61O4MLT&qid=1653308124&sprefix=ba%2Caps%2C283&ref=sr_pg_1'
def retrieveSiteData(link):
site_info = requests.get(link)
status = site_info.status_code
if status == 200:
html_text = site_info.text
return html_text
else:
retrieveSiteData(link)
content = retrieveSiteData(url)
print(content)