1

I have a lot of software running on an embedded system with ruby 1.8.6. One script queries the national weather service and a few days ago it stopped working: I started getting <H1>Access Denied</H1>. This is a public interface and there's no API key.

I get the same error when I run this script using ruby 1.8.6 on my development machine. When I run it with ruby 1.9, the error doesn't happen.

Evidently the NWS modified their HTTP protocol in some way last week that is incompatible with Net:HTTP in ruby 1.8.6

The URL is "http://graphical.weather.gov/xml/sample_products/browser_interface/ndfdXMLclient.php?lat=37&lon=-105&product=time-series&begin=&end=&sky=sky&maxt=maxt&mint=mint&Submit=Submit"

I get a good response when I use curl.

The code is: begin xml = Net::HTTP.get_response(URI.parse(url)).body rescue

I have RestClient 1.6 and tried substituting that for Net:HTTP but it just uses Net:HTTP internally and I get a 403 Forbidden.

Does anyone have any idea what could have changed in the protocol and a way to work around it?

I think I may be able to get updated firmware from the system manufacturer but I hesitate to go that route because it will force rewriting of many rather fragile hacks.

chetstone
  • 650
  • 1
  • 9
  • 19
  • 1
    as an alternative, host your own mini api that proxiesnws data that you know works with your old stack – Keith Nicholas Feb 08 '17 at 01:33
  • try taking a traffic capture (eg tcpdump/Wireshark) on your dev machine, of working and not working connection. if difference won't be evident from there, you can try posting the http requests comparison here – Alien_AV Feb 08 '17 at 02:02
  • Would you mind adding a [mcve]? Just some code that I can run to repro the issue? – thesecretmaster Feb 08 '17 at 02:25
  • @Alien_AV that's the ticket. Net:HTTP 1.8.6 was not including a User-Agent field. I added one like [this](http://stackoverflow.com/a/5798178/1404185) and that fixed the problem. Thanks! (Maybe you can post your comment as an answer?) – chetstone Feb 08 '17 at 03:27

1 Answers1

2

According to your traffic capture analysis, the issue is that 1.8.6 doesn't add HTTP User-Agent field by default. One way to add it is:

http = Net::HTTP.new("your.site.com", 80)
req = Net::HTTP::Get.new("/path/to/the/page.html", {'User-Agent' => 'your_agent'})
response = http.request(req)
puts response.body
Alien_AV
  • 345
  • 1
  • 9