1

Since a few weeks ago I'm not able to download Yahoo finance data any more:

$ wget -O GLD.USA_20170612.txt --no-check-certificate http://chart.finance.yahoo.com/table.csv?s=GLD&a=2&b=1&c=2017&d=11&e=30&f=2017&ignore=.csv
--2017-06-12 12:21:28--  http://gld.usa_20170612.txt/
Resolving gld.usa_20170612.txt (gld.usa_20170612.txt)... failed: No address associated with hostname.
wget: unable to resolve host address ‘gld.usa_20170612.txt’
--2017-06-12 12:21:28--  http://chart.finance.yahoo.com/table.csv?s=GLD&a=2&b=1&c=2017&d=11&e=30&f=2017
Resolving chart.finance.yahoo.com (chart.finance.yahoo.com)... 87.248.114.12, 87.248.116.11, 87.248.116.12, ...
Connecting to chart.finance.yahoo.com (chart.finance.yahoo.com)|87.248.114.12|:80... connected.
HTTP request sent, awaiting response... 301 Moved Permanently
Location: https://chart.finance.yahoo.com/table.csv?s=GLD&a=2&b=1&c=2017&d=11&e=30&f=2017 [following]
--2017-06-12 12:21:28--  https://chart.finance.yahoo.com/table.csv?s=GLD&a=2&b=1&c=2017&d=11&e=30&f=2017
Connecting to chart.finance.yahoo.com (chart.finance.yahoo.com)|87.248.114.12|:443... connected.
HTTP request sent, awaiting response... 404 Not Found
2017-06-12 12:21:28 ERROR 404: Not Found.

They have changed something. I picked up a new URL from the site, but it still does not work:

$ wget https://query1.finance.yahoo.com/v7/finance/download/GLD?period1=1494584558&period2=1497262958&interval=1d&events=history&crumb=GGHpj6ucgIy
--2017-06-12 12:27:24--  https://query1.finance.yahoo.com/v7/finance/download/GLD?period1=1494584558
Resolving query1.finance.yahoo.com (query1.finance.yahoo.com)... 87.248.116.11, 87.248.116.12, 87.248.114.11, ...
Connecting to query1.finance.yahoo.com (query1.finance.yahoo.com)|87.248.116.11|:443... connected.
HTTP request sent, awaiting response... 401 Unauthorized

Username/Password Authentication Failed.

I don't understand why it mentions Username and Password. When I click the Download button at https://uk.finance.yahoo.com/quote/GLD/history?p=GLD, it does not ask for the Username and Password. So it seems that data can be downloaded without a subscription.

If someone knows a proper wget implementation for downloading Yahoo finance data, please share it here.

...

Update:

Thanks to the current replies, I was hinted that "cookies" might be involved here. While searching for the similar problem with cookies in the keywords, I found the following threads:

Yahoo Finance Historical data downloader url is not working

Yahoo Finance URL not working

Unfortunately this is too complex for me... I would appreciate a little help in making this work.

Boris L.
  • 936
  • 2
  • 12
  • 28
  • _"I don't understand why it mentions Username and Password. When I click the Download button at https://uk.finance.yahoo.com/quote/GLD/history?p=GLD, it does not ask for the Username and Password"_ - then I suggest you open that download link in a _private_ browser tab for a change, and find the explanation ... – CBroe Jun 12 '17 at 12:05
  • You're missing a cookie, you could check what cookies you send when you click the download button. I suspect Yahoo doesn't want you accessing their finance data without using their own API. When you do that you are bypassing their ads. – matt Jun 12 '17 at 12:16
  • If I could identify which cookie, and store it offline... How to instruct `wget` to use the cookie? – Boris L. Jun 12 '17 at 13:42
  • better of using curl then... but it is possible with wget, checkout [this post](https://stackoverflow.com/a/4272783/777850) or google some more. – giorgio Jun 12 '17 at 14:46

1 Answers1

4

In order to deal with Yahoo cokies I have written the following code:

#!/usr/bin/sh

symbol=$1
today=$(date +%Y%m%d)
tomorrow=$(date --date='1 days' +%Y%m%d)

first_date=$(date -d "$2" '+%s')
last_date=$(date -d "$today" '+%s')

wget --no-check-certificate --save-cookies=cookie.txt https://finance.yahoo.com/quote/$symbol/?p=$symbol -O crumb.store

crumb=$(grep 'root.*App' crumb.store | sed 's/,/\n/g' | grep CrumbStore | sed 's/"CrumbStore":{"crumb":"\(.*\)"}/\1/')

wget --no-check-certificate --load-cookies=cookie.txt "https://query1.finance.yahoo.com/v7/finance/download/$symbol?period1=$first_date&period2=$last_date&interval=1d&events=history&crumb=$crumb" -O $symbol.csv

rm cookie.txt crumb.store

Usage example:

$ sh stockdownload_yahoo.sh QQQ 20170508

Examples supported date formats:

$ date -d 20170328 +%s
1490652000
$ date -d 2017-03-28 +%s
1490652000
$ date -d "Mar 28 2017" +%s
1490652000
Boris L.
  • 936
  • 2
  • 12
  • 28