Remote server (403) Forbidden error while using WebClient

Question

I am trying to query some URL using WebClient.

I have a collection which I loop through to get the QueryString value, and build final URL, and then pass it on to client.

The very first time it gets executed well, I get proper response, however, when it goes in loop second time, I get the error:

System.Net.WebException --> The remote server returned an error: (403) Forbidden.

If I get response for very first time. Then I should get for rest of the collection too.

Any clue why? What I might be missing?

Below is the code snippet I am using.

 using(System.IO.StreamWriter file = new System.IO.StreamWriter(@"C:\sample.text")) {
    foreach(var f in fileCollections) {
        strFinalURL = string.Empty;

        strFinalURL = "someURL" + f; // f can be considered as querystring param value

        try {
            using(var client = new WebClient()) {

                test = client.DownloadString(strFinalURL);
                if (!test.Contains("somecondition")) {
                    file.WriteLine("");
                }
            }
        } catch (System.Exception ex) {
            Console.WriteLine(ex.Message);
        }
    }
}

Are all URLs from the same domain? The URL that returns 403, is it accessible via a browser? — Tasos K., Oct 31 '13 at 20:25
@codingstill yes, all URLs are from same domain and accessible via brower — Amit, Oct 31 '13 at 20:30
If everything you said is correct, then I suspect server throttling is playing a part. What happens if you slow things down a bit with a strategically placed `Thread.Sleep`? — spender, Oct 31 '13 at 20:34
Download [Fiddler](http://fiddler2.com/) and send an HTTP request using fiddler and look at the data being received and sent. — Icemanind, Oct 31 '13 at 20:55
@spender, Yup Throttling is playing role here.. when I am slowing down the process.. it works... Thanks — Amit, Oct 31 '13 at 20:56

score 17 · Answer 1 · answered Oct 31 '13 at 20:42

17

Some web servers can block requests based on the user agent string they provide. In your case, you send an empty string as a user agent. Try to add the user agent of a browser, any browser would be fine.

For example:

client.Headers.Add("user-agent", " Mozilla/5.0 (Windows NT 6.1; WOW64; rv:25.0) Gecko/20100101 Firefox/25.0");

For a list of user agents, you can click here.

answered Oct 31 '13 at 20:42

Tasos K.

7,979
7
39
63

2

This fixed it for me, would never have thought of that! – Simon Feb 28 '16 at 19:04

score 1 · Accepted Answer · answered Oct 31 '13 at 21:14

1

If everything you said is correct, then I suspect server throttling is playing a part. Try adding a Thread.Sleep in your loop to slow things down.

answered Oct 31 '13 at 21:14

spender

117,338
33
229
351

Remote server (403) Forbidden error while using WebClient

2 Answers2

Linked