21

Opening a public page from browser works fine.

Downloading same page using WebClient throws - (403) Forbidden.

What is going on here ?

Here is quick copy/paste example (used on console app) to specific page on web:

try
{
    WebClient webClient = new WebClient();
    string content = webClient.DownloadString("http://he.wikisource.org/wiki/%D7%A9%D7%95%D7%9C%D7%97%D7%9F_%D7%A2%D7%A8%D7%95%D7%9A_%D7%90%D7%95%D7%A8%D7%97_%D7%97%D7%99%D7%99%D7%9D_%D7%90_%D7%90");
}
catch (Exception ex)
{
    throw;
}
John Saunders
  • 160,644
  • 26
  • 247
  • 397
dzolnjan
  • 1,243
  • 4
  • 14
  • 26
  • how often are you calling this url? you could be throttled. – Nix May 08 '10 at 13:37
  • Not much I think, few times from browsers before trying with webclient. I have been downloading other pages from same site much more by now and they work fine every time. So it looks like this page only is throwing error?! – dzolnjan May 08 '10 at 13:45
  • 1
    The page may be looking at the user agent header, and bouncing you because you don't provide one. – John Saunders May 08 '10 at 13:59

2 Answers2

64

I've just tried it with Fiddler running to see the response and it returns the following notice with the status code.

Scripts should use an informative User-Agent string with contact information, or they may be IP-blocked without notice.

This works.

    WebClient webClient = new WebClient();
    webClient.Headers.Add("user-agent", "Only a test!");

    string content = webClient.DownloadString("http://he.wikisource.org/wiki/%D7%A9%D7%95%D7%9C%D7%97%D7%9F_%D7%A2%D7%A8%D7%95%D7%9A_%D7%90%D7%95%D7%A8%D7%97_%D7%97%D7%99%D7%99%D7%9D_%D7%90_%D7%90");
Martin Smith
  • 438,706
  • 87
  • 741
  • 845
  • 1
    Thanks for this - it didn't solve it for me, but you got me on a track. As a word of warning to those that follow, check your own security implementation first. I had forgotten to grant my user the rights to perform downloads and was receiving (correctly) a 403 message. Nothing to do with the IIS server, the client, the user-agent or anything. – The Senator Jun 08 '15 at 15:14
  • @TheSenator thanks, in my case it was IIS IP address restrictions. – joym8 Oct 03 '17 at 14:59
3

Check if the server, you are trying to access is set up to use improved TLS protocol. Make sure to add this to Global.asax.cs

ServicePointManager.SecurityProtocol = SecurityProtocolType.Tls12;
Hrvoje Matic
  • 1,207
  • 15
  • 12