WebClient 403 Forbidden

Question

I can download this by hand in IE.

http://scholar.google.com/scholar.ris?q=info:j8ymU9rzMsEJ:scholar.google.com/&output=citation&hl=zh-CN&as_sdt=2000&oe=GB&ct=citation&cd=0

But, using follow code

WebClient client = new WebClient();
client.DownloadFile(address, filename);

Show Exception: 403 Forbidden

What's wrong? How can I do that?

others

http://scholar.google.com/scholar.ris?q=info:sskrpr5jlLwJ:scholar.google.com/&output=citation&hl=zh-CN&as_sdt=2000&oe=GB&ct=citation&cd=1

Answer 1

Just add a simple line before you make your download:

string url = ... 
string fileName = ...

WebClient wb = new WebClient();
wb.Headers.Add("User-Agent: Other");   //that is the simple line!
wb.DownloadFile(url, fileName);

That's it.

Answer 2

403 may also be caused by TLS issues. To verify, you should check the text of the WebException.Response object.

     catch (WebException ex)
     {
        if (ex.Response != null)
        {
           var response = ex.Response;
           var dataStream = response.GetResponseStream();
           var reader = new StreamReader(dataStream);
           var details = reader.ReadToEnd();
        }
     }

If it is TLS then try adding this to your code to force TLS1.2.

For .net4:

ServicePointManager.SecurityProtocol = (SecurityProtocolType)3072;

For .net4.5 or later:

ServicePointManager.SecurityProtocol = SecurityProtocolType.Tls12;

Answer 3

I had this problem trying to download an image from a SharePoint site url. In my case setting the user-agent to Other or blank in the header wasn't enough, I had to set the user-agent as follows instead:

client.Headers.Add("user-agent", " Mozilla/5.0 (Windows NT 6.1; WOW64; rv:25.0) Gecko/20100101 Firefox/25.0");

That solution came from this answer.

Answer 4

You need to set appropriate http headers before calling your DownloadFile method.

WebClient webClient = new WebClient();
webClient.Headers.Add("???", "???");
webClient.Headers.Add("???", "???");
webClient.Headers.Add("???", "???");
webClient.DownloadFile(address, filename);

To put correct values instead of these question marks might be tricky. You will need to download Fiddler or some other program or webbrowser extension to reveal what http headers are being sent to Google by your webbrowser and basically replicate the same request in your program.

Answer 5

This is what happened with me:

I was trying to download a (public) .xls file (via DownloadFile method) which was getting downloaded comfortably from all browsers.

After trying and struggling with all answers (but no luck), I finally opened the Stack and noticed something odd (refer screenshot).

Although, the file was getting downloaded via http in browser but it was giving 403 error via DownloadFile method.

Finally, I just changed the URL from http://something to https://something and it worked fine.

Hope this helps!

Answer 6

I get a 403 in IE, I guess you need to be logged in to retrieve the resource. Your browser may have the credentials cached but your app isn't designed to log you in. Or are you logged in to Google in your browser - try logging out and see if you still have access....

Answer 7

The key to solving this for me was to do the request once via code, a second time in the browser, log both requests with Fiddler and ensure the headers match up.

I ended up having to add headers for:

• Accept
• Accept-Encoding
• Accept-Language
• User-Agent
• Upgrade-Insecure-Requests

I hope this helps people in the future.

Answer 8

I ran into the same issue trying to download a file on an Amazon 3S url. I blogged about it here: http://blog.cdeutsch.com/2010/11/net-webclient-403-forbidden-error.html

The final solution I used was found here: GETting a URL with an url-encoded slash