Website giving error page when trying to get webpage using python-request

Asked Apr 06 '20 at 16:07

Active Apr 06 '20 at 19:39

Viewed 304 times

I am trying to get the HTML code from a Yahoo Finance page in order to do some web scraping stuff and create tables from data, etc.

The issue is that any website I try this with is redirecting me to the error/bot page so I am not being able to get the code or the actual page I need, I am using the Pycharm Python Editor and the request module

import requests
from bs4 import BeautifulSoup
url = 'https://finance.yahoo.com/cryptocurrencies'
page = requests.get(url)
soup = BeautifulSoup(page.text, 'lxml')
print(soup)

I can tell I am on the wrong page since the HTML has stuff like robot and error in it Because of this I am unable to move forward in my project.

Note: it works with some websites just fine but with some others, I automatically get sent to the error page

edited Apr 06 '20 at 19:39

asked Apr 06 '20 at 16:07

CSE_Random

1

Some websites might block incoming requests if the HTTP header User-Agent is missing or does not correspond to a known web browser. – Buddhima Gamlath Apr 06 '20 at 16:14
Ok, so can you walk me in fixing that? – CSE_Random Apr 06 '20 at 17:24
Try this for example: https://stackoverflow.com/questions/27652543/how-to-use-python-requests-to-fake-a-browser-visit – Buddhima Gamlath Apr 07 '20 at 22:57

Website giving error page when trying to get webpage using python-request

0 Answers0