scraping with request with bs4

Question

I am try to scrape this page getting a 406 error. what am i doing wrong please?

import requests
from bs4 import BeautifulSoup

url="http://chucknorrisfacts.net/facts"

response = request.get(url)
print(response)

this is not [ask] a question, but see https://stackoverflow.com/questions/56101612/python-requests-http-response-406, you probably need a user agent — coderoftheday, Nov 27 '20 at 14:03

score 0 · Accepted Answer · answered Nov 27 '20 at 14:23

add the headers parameter:

import requests
from bs4 import BeautifulSoup

url="http://chucknorrisfacts.net/facts"
headers = {'user-agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/87.0.4280.66 Safari/537.36'}

response = requests.get(url,headers=headers)
print(response)

scraping with request with bs4

1 Answers1