1

I want to pull data from here.

Utlimately I want to create an app which will analyse the yield curve data, create interactive graphs and publish economic conclusions based on movement of interest rates. I'll do this using php and javascript.

It is clear from data.gov that the data is public. They reference their license which says the material can be shared, copied and redistributed in any medium or format; it can be adapted, remixed, transformed, and built upon for any purpose, even commercially.

Given that my cross origin requests are blocked because CORS header 'Access-Control-Allow-Origin' missing (Please see my prior question for details), is web scraping appropriate? Where can I find some definitive law/rules on this subject? Are there different web scraping methods I should research? From a bit of googling, it seems that web scraping is controversial.

Please could someone explain.

Community
  • 1
  • 1
DVCITIS
  • 1,067
  • 3
  • 16
  • 36
  • 1
    Running `curl "http://www.treasury.gov/resource-center/data-chart-center/interest-rates/pages/XmlView.aspx?data=yieldyear&year=2015" > test.xml` in my terminal works fine -- so maybe you should grab the content via the server-side (PHP). – arthurakay Nov 11 '15 at 19:40
  • Thanks for the comment i was successful with curl also and have made some headway with php. Running into some new issues if you have any suggestions. Created a new Q here: http://stackoverflow.com/questions/33663864/php-not-parsing-xml-in-expected-format – DVCITIS Nov 12 '15 at 03:45
  • Still interested to learn more about web scraping if anyone has some info on that front though? – DVCITIS Nov 12 '15 at 04:09

0 Answers0