Error in web-Scraping Code Using BeautifulSoup

Question

I want to get data from https://www.cvedetails.com/vulnerability-list/vendor_id-26/product_id-32238/Microsoft-Windows-10.html from page 1 to the last page while it is sorted by "CVE Number Ascending" The data I want to retrieve in CSV format is everything in the table header and the table data

I have been trying out a few codes but it doesn't seem to work and I'm kind of desperate now

https://youtu.be/XQgXKtPSzUI

the place I try to learn from

Any help would be appreciated

I asked this once before The replies I got were great But it doesn't seem to get what I need and I am confused about how this works And more so because of how weird the sources code for the website is

#!/usr/bin/env python3
import bs4 # Good HTML parser
from urllib.request import urlopen as uReq # Helps with opening URL
from bs4 import BeautifulSoup as soup

# The target URL
my_url = 'https://www.cvedetails.com/vulnerability-list.php?vendor_id=26&product_id=32238&version_id=&page=1&hasexp=0&opdos=0&opec=0&opov=0&opcsrf=0&opgpriv=0&opsqli=0&opxss=0&opdirt=0&opmemc=0&ophttprs=0&opbyp=0&opfileinc=0&opginf=0&cvssscoremin=0&cvssscoremax=0&year=0&month=0&cweid=0&order=2&trc=851&sha=41e451b72c2e412c0a1cb8cb1dcfee3d16d51c44'

# Check process
# print(my_url)

# Open a connection and grab the webpage and downloads it
uClient = uReq(my_url)

# Save the webpage into a variable
page_html = uClient.read()

# Close the internet connection from uclient
uClient.close()

# Calling soup to parse the html with html parser and saving it to a variable
page_soup = soup(page_html,"html.parser")

print(page_soup.h1)

This is the error code

Traceback (most recent call last):
  File "./Testing3.py", line 21, in <module>
    uClient = uReq(my_url)
  File "/usr/lib/python3.6/urllib/request.py", line 223, in urlopen
    return opener.open(url, data, timeout)
  File "/usr/lib/python3.6/urllib/request.py", line 532, in open
    response = meth(req, response)
  File "/usr/lib/python3.6/urllib/request.py", line 642, in http_response
    'http', request, response, code, msg, hdrs)
  File "/usr/lib/python3.6/urllib/request.py", line 570, in error
    return self._call_chain(*args)
  File "/usr/lib/python3.6/urllib/request.py", line 504, in _call_chain
    result = func(*args)
  File "/usr/lib/python3.6/urllib/request.py", line 650, in http_error_default
    raise HTTPError(req.full_url, code, msg, hdrs, fp)
urllib.error.HTTPError: HTTP Error 403: Forbidden

Possible duplicate of [HTTP error 403 in Python 3 Web Scraping](https://stackoverflow.com/questions/16627227/http-error-403-in-python-3-web-scraping) — Rahul Agarwal, May 15 '19 at 11:30
Some Websites do not allow to scrape data from bot. So, you should send some headers to bypass it. — Apeal Tiwari, May 15 '19 at 11:45

score 0 · Accepted Answer · answered May 15 '19 at 11:36

To avoid this error, you need to supply the user agent through the header in the request.

Try modifying your script as:

#!/usr/bin/env python3
import bs4
from urllib.request import urlopen as uReq, Request
from bs4 import BeautifulSoup as soup

#bs4 is a good html parser
#urllib.request helps with opening the url


#setting the target url
my_url = 'https://www.cvedetails.com/vulnerability-list.php?vendor_id=26&product_id=32238&version_id=&page=1&hasexp=0&opdos=0&opec=0&opov=0&opcsrf=0&opgpriv=0&opsqli=0&opxss=0&opdirt=0&opmemc=0&ophttprs=0&opbyp=0&opfileinc=0&opginf=0&cvssscoremin=0&cvssscoremax=0&year=0&month=0&cweid=0&order=2&trc=851&sha=41e451b72c2e412c0a1cb8cb1dcfee3d16d51c44'

hdr = {'User-Agent': 'Mozilla/5.0'}
req = Request(my_url,headers=hdr)
page = uReq(req)
page_soup = soup(page)

print(page_soup.h1)

score 0 · Answer 2 · answered May 15 '19 at 11:38

Instead of urllib why don't you directly use requests module. Try this code

import requests
from bs4 import BeautifulSoup as soup
my_url = 'https://www.cvedetails.com/vulnerability-list.php?vendor_id=26&product_id=32238&version_id=&page=1&hasexp=0&opdos=0&opec=0&opov=0&opcsrf=0&opgpriv=0&opsqli=0&opxss=0&opdirt=0&opmemc=0&ophttprs=0&opbyp=0&opfileinc=0&opginf=0&cvssscoremin=0&cvssscoremax=0&year=0&month=0&cweid=0&order=2&trc=851&sha=41e451b72c2e412c0a1cb8cb1dcfee3d16d51c44'

page_html = requests.get(my_url).text

page_soup = soup(page_html,"html.parser")

print(page_soup.h1)

output:

<h1>
<a href="//www.cvedetails.com/vendor/26/Microsoft.html" title="Details for Microsoft">Microsoft</a> » <a href="//www.cvedetails.com/product/32238/Microsoft-Windows-10.html?vendor_id=26" title="Product Details Microsoft Windows 10">Windows 10</a> : Security Vulnerabilities
</h1>

Error in web-Scraping Code Using BeautifulSoup

2 Answers2