1

Valid URL fails requests.get

https://finance.yahoo.com/quote/AAPL/analysis?p=AAPL is a valid URL as is not redirects

DOES NOT WORK

import requests

url6 = 'https://finance.yahoo.com/quote/AAPL/analysis?p=AAPL'
r = requests.get(url6)

returns

False

404

[]

or more simply

requests.get('https://finance.yahoo.com/quote/AAPL/analysis?p=AAPL')

<Response [404]>

WORKS this for example (different page on same source)

requests.get('https://finance.yahoo.com/quote/AMG?p=AMG')

returns True 200

urbanscribe
  • 11
  • 3
  • 1
    You have to add headers to your request. Does this answer your question? [Sending "User-agent" using Requests library in Python](https://stackoverflow.com/questions/10606133/sending-user-agent-using-requests-library-in-python). – mx0 Nov 23 '22 at 15:58
  • 1
    They probably have something in place to stop webscraping, try setting a user-agent header – Holloway Nov 23 '22 at 15:58

2 Answers2

0

I added headers to your request. More specifically, I added the user agent.

import requests

url6 = 'https://finance.yahoo.com/quote/AAPL/analysis?p=AAPL'
headers={'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/102.0.0.0 Safari/537.36'}

r = requests.get(url6, headers=headers)
Cezar Crintea
  • 301
  • 4
  • 7
0

Seems that adding user-agent helps

r = requests.get(url8, headers={'User-Agent': 'Custom'})
urbanscribe
  • 11
  • 3