0

Before using BeautifulSoup on html text of this website, I'm trying to get data using 'requests' library.

But, the output sometimes returns html text and other times it returns 'None'. Can I somehow modify the code to always return html text of the website?

from bs4 import BeautifulSoup
import requests

url = 'https://www.amazon.in/s?k=bags&crid=2M096C61O4MLT&qid=1653308124&sprefix=ba%2Caps%2C283&ref=sr_pg_1'

def retrieveSiteData(link):
    site_info = requests.get(link)
    status = site_info.status_code

    if status == 200:
        html_text = site_info.text
        return html_text
    else:
        retrieveSiteData(link)

content = retrieveSiteData(url)
print(content)

Shivam
  • 19
  • 4

0 Answers0