1

I am trying to scrape this website: https://www.ralphlauren.com/men?webcat=men but I don't get the HTML of the link, instead I get an HTML which states Access to this page has been denied.

I tried using user agent string in the header as suggested here solution: Scraper in Python gives "Access Denied"

but I still get the same error.

Current Output:

  <!DOCTYPE html>
<html lang="en">
<head>
<meta charset="utf-8"/>
<meta content="width=device-width, initial-scale=1" name="viewport"/>
<title>Access to this page has been denied.</title>
<link href="https://fonts.googleapis.com/css?family=Open+Sans:300" rel="stylesheet"/>

My Code:

from bs4 import BeautifulSoup as soup
import requests
url = "https://www.ralphlauren.com/men?webcat=men"

def make_soup(url):
  header = {"User-Agent":"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/103.0.0.0 Safari/537.36"}
  page = requests.get(url,headers=header)
  page_soup = soup(page.content, 'lxml')
  return page_soup 

print(make_soup(url))
user13174343
  • 235
  • 2
  • 10
  • 1: I can't reproduce, I do get back a full response. 2: the page is loaded _dynamically_ so, either way, you probably won't be able to get get the desired response. See [Web-scraping JavaScript page with Python](https://stackoverflow.com/questions/8049520/web-scraping-javascript-page-with-python) – MendelG Jul 05 '22 at 11:55
  • perhaps access is closed for your country, try using a proxy – Sergey K Jul 05 '22 at 12:02

1 Answers1

1

The website is in cloudflare protection. You can use cloudscraper in lieu of requests

from bs4 import BeautifulSoup
import cloudscraper
scraper = cloudscraper.create_scraper(delay=10,   browser={'custom': 'ScraperBot/1.0',})
url = 'https://www.ralphlauren.com/men?webcat=men'
req = scraper.get(url)
print(req)
soup = BeautifulSoup(req.text,'lxml')

Output:

<Response [200]>
Md. Fazlul Hoque
  • 15,806
  • 5
  • 12
  • 32