2

I am trying to scrape the historical data from Yahoo finance stocks. I found some code from the internet and modified it. It worked at first, but now I cannot scrape the crumb from yahoo. Can anyone advise how to fix this?

The first time I scraped yahoo, it scraped 20 sets of data and stopped working. I started it again and it did't work

The relevant code is attached below

def _get_crumbs_and_cookies(stock):
    """
    get crumb and cookies for historical data csv download from yahoo finance

    parameters: stock - short-handle identifier of the company 

    returns a tuple of header, crumb and cookie
    """

    url = 'https://finance.yahoo.com/quote/{}/history'.format(stock)
    with requests.session():
        header = {'Connection': 'keep-alive',
                   'Expires': '-1',
                   'Upgrade-Insecure-Requests': '1',
                   'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; WOW64) \
                   AppleWebKit/537.36 (KHTML, like Gecko) 
                   Chrome/54.0.2840.99 Safari/537.36'
                   }

        website = requests.get(url, headers=header)
        soup = BeautifulSoup(website.text, 'lxml')
        crumb = re.findall('"CrumbStore":{"crumb":"(.+?)"}', str(soup))

    return (header, crumb[0], website.cookies)

And the entire original code can be found at: https://maikros.github.io/yahoo-finance-python/

Thanks in advance for your help

toti08
  • 2,448
  • 5
  • 24
  • 36
maggiemee
  • 21
  • 2
  • 1
    Welcome to Stack Overflow. It sounds like Yahoo Finance may have blocked you, or you made a change which broke the scraping. Try again later and see if it works then (blocking is often temporary). – Mikkel Sep 05 '18 at 19:48

2 Answers2

0

I think you should take a look at this post here: Yahoo Finance API / URL not working: Python fix for Pandas DataReader

It looks like yahoo finance has discontinued their historical data :(

You can try google ! However, Google is not really a viable option because they adjust prices for splits, but not for dividends.

Hope this helps!

Julian Silvestri
  • 1,970
  • 1
  • 15
  • 33
-1

I've been looking for a finance API like a week ago. I figured, that either Yahoo or Google should supply a good one, but I had to find out, that both have been discontinued.

For now I'm using WorldTradingData which is free for 250 requests a day. I don't know which scale you will be working on, but if you need this for an App they do also offer paid subscriptions with way more requests possible. I found their customer support to be very nice and helpful, as I found out that some live quotes were wrong and they only a day later already switched the location they get their data from and everything was great again!

I hope this gets you where you want to be!

Lukas
  • 33
  • 1
  • 10