Current Observation feed from weather.gov forbidden (403)

Question

Weather.gov Current Observation feeds have suddenly begun to fail for all requests from an HTTPClient, and likewise I've observed that many websites across the internet that use AJAX to make calls to weather.gov are also failing.

The result of all calls to weather.gov current observation feeds, e.g. http://w1.weather.gov/xml/current_obs/TAPA.xml, return a 403. Said URL resolves properly in a browser.

score 25 · Accepted Answer · answered Sep 17 '15 at 22:40

Contacting weather.gov resulted in a really fast response, which was:

Applications accessing resources on weather.gov now need to provide a User-Agent header in any HTTP request. Requests without a user agent are automatically blocked. We have implemented this usage policy due to a small number of clients utilizing resources far in excess of what most would consider reasonable.

We recommend providing a user agent string in the following format:

ApplicationName/vX.Y (http://your.app.url/; contact.email@example.com)

This will both uniquely identify your application and allow us to contact you and work with you if we observe abnormal application behavior that may result in a block.

Please feel free to email us back if you continue to have issues after verifying that your application is sending the proper headers.

Thanks for using weather.gov.

=======

Here's a snip of C# code. The key thing is that you need to create the request object then append a custom User-Agent string to it before making the call.

...
var request = new HttpRequestMessage(HttpMethod.Post, httpClient.BaseAddress.AbsoluteUri);
request.Headers.Add("User-Agent", "MyApplication/v1.0 (http://foo.bar.baz; foo@bar.baz)");
var httpResponse = httpClient.SendAsync(request).Result;
...

Hope this helps folks. Cheers

This is awesome when simply emailing someone yields the right response :) — Julien Genestoux, Sep 18 '15 at 08:06
@JulienGenestoux I got the response in 3 minutes. Suffice it to say I was astounded that a government agency got back to me that fast. Color me impressed. — Jim Speaker, Sep 18 '15 at 17:21
Thanks. Six years later (almost to the day), and this still helped me. — John Pasquet, Sep 14 '21 at 13:07

R-TEC Guru · Answer 2 · 2015-10-02T02:26:54.473

That is great but you cant set "User-Agent" when you are using an XMLDocument and calling Load() Like this (this used to work):

        XmlDocument doc = new XmlDocument();
        string stationName = @"http://w1.weather.gov/xml/current_obs/PASC.xml";
        doc.Load(stationName);  // exception 403 occurs here now

Instead now you need to perform a GET and then set the User-Agent to your company or email and then use the XmlDocument like:

       XmlDocument doc = new XmlDocument();

       string stationName = @"http://w1.weather.gov/xml/current_obs/PASC.xml";

        HttpWebRequest request = (HttpWebRequest)WebRequest.Create(stationName);

        request.Method = "GET";
        request.ContentType = "application/x-www-form-urlencoded";
        request.UserAgent = "MyApplication/v1.0 (http://foo.bar.baz; foo@bar.baz)";
        request.Accept = "Accept=text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8";

        HttpWebResponse response = (HttpWebResponse)request.GetResponse();
        Stream resStream = response.GetResponseStream();

        doc.Load(resStream);
        try
        {

            XmlNodeList list = doc.GetElementsByTagName("temp_f");
            if (list.Count > 0)
            {
                float fTemperature = float.Parse(list.Item(0).InnerText);

            }
        }

True enough. They've locked it down too much to use the old method. — Jim Speaker, Oct 05 '15 at 21:22

score 2 · Answer 3 · answered Jan 23 '16 at 03:05

2

On the off chance your finding this from Étude 12-1 from Études for Elixir, here's what ended up working for me

  s_url = "http://w1.weather.gov/xml/current_obs/" <> weather_station <> ".xml"
 :httpc.request(:get, { to_char_list(s_url),
                        [ {'User-Agent','ErlangEtudes/v1.0 (http://example.org;me@example.org)' } ] 
                      },[],[] )

answered Jan 23 '16 at 03:05

pwan

2,894
2
24
38

Excellent. Thank you. – Jim Speaker Jan 23 '16 at 18:19

score 1 · Answer 4 · answered Oct 29 '15 at 19:47

I had the same problem but was using PowerShell. Here is a way to set User Agent by using HttpWebRequest.

$weather_url = "http://w1.weather.gov/xml/current_obs/KLNK.xml"
$request = [System.Net.HttpWebRequest]::Create($weather_url)
$request.UserAgent = " {Enter your agent} "
$response = $request.GetResponse()

$doc = New-Object System.Xml.XmlDocument    
$doc.Load($response.GetResponseStream())

$temp_f = [int] $doc.current_observation.temp_f

score 1 · Answer 5 · edited Nov 02 '21 at 10:23

1

This answer helped me but actually did not work. I still received the error response.

I changed the Post to a Get in this line:

var request = new HttpRequestMessage(HttpMethod.Post...

To:

var request = new HttpRequestMessage(HttpMethod.Get...

Then, it worked fine.

edited Nov 02 '21 at 10:23

Yu Hao

119,891
44
235
294

answered Nov 01 '21 at 21:02

Lloyd S Turley

31
2

1

Your answer could be improved with additional supporting information. Please [edit] to add further details, such as citations or documentation, so that others can confirm that your answer is correct. You can find more information on how to write good answers [in the help center](/help/how-to-answer). – Community Nov 01 '21 at 22:16
Interesting. The product that I have in Production is working properly with a POST request with the adjustments based on my answer. This solution has been in place without adjustment in production since the day that I wrote up this Question/Answer. There must be some other issue with your code. – Jim Speaker Nov 03 '21 at 13:46
https://fishing.smarttripmap.com/ Weather features are working properly. The Conditions tab on the right uses this feed. – Jim Speaker Nov 03 '21 at 13:50

Current Observation feed from weather.gov forbidden (403)

5 Answers5

Linked

Related