HTTP Request only works in browser

Question

I have the following GET request that returns HTML for a login form, indicating that my authentication, i.e. the credentials are wrong. When I am authenticated in a browser session, and manually request the same URL, I get the expected empty XML document as a response. What am I missing?

    var getRequest = WebRequest.Create("http://frulo.com/v1/company/subscribers.xml") as HttpWebRequest;
    getRequest.Credentials = new NetworkCredential("user@company.net", "password");
    using (var response = getRequest.GetResponse() as HttpWebResponse)
    {
        var sr = new StreamReader(response.GetResponseStream());
        Response.Write(sr.ReadToEnd());
    }

score 0 · Answer 1 · answered Oct 19 '09 at 21:57

0

They may be blocking your program's user agent to prevent this sort of scraping.

answered Oct 19 '09 at 21:57

ceejayoz

176,543
40
303
368

Yeah, try inputting a different User Agent string: http://msdn.microsoft.com/en-us/library/system.net.httpwebrequest.useragent.aspx – NickAldwin Oct 19 '09 at 22:05

score 0 · Accepted Answer · edited May 23 '17 at 10:34

0

You already know the answer: When you are authenticated in a browser session you get the correct response. That means that you are not authenticated when using the WebRequest.

The credentials that you provide are used for HTTP authentication but your web site uses most likely some sort of HTML forms-based authentication.

To solve the problem you will have to use the same authentication mechanism that the web application does. This might be cookie based or a session id might be transmitted as a POST or GET parameter along with each request. Without knowing further details about the web site it's difficult to provide more help though.

The following question is related and most likely of help for you:

C# Login to Website via program

edited May 23 '17 at 10:34

Community

1
1

answered Oct 19 '09 at 22:20

Dirk Vollmar

172,527
53
255
316

Hmm, and this is a what is euphemistically called a documented REST API. The documentation is an example with only the actual GET request from my code, a one line example. – ProfK Oct 19 '09 at 22:25
Well, if you don't get good documentation you probably will have to do some reverse engineering. Install Fiddler (or another proxy you like) and monitor the HTTP traffic that is sent over the wire. – Dirk Vollmar Oct 19 '09 at 22:36

score 0 · Answer 3 · answered Oct 22 '09 at 04:11

Using the Web Request Proxy property might help in some circumstances. I have inserted a line of code with a comment by MSDN.

   var getRequest = WebRequest.Create("http://frulo.com/v1/company/subscribers.xml") as HttpWebRequest;

    //MSDN states: Returns a proxy configured with the Internet Explorer settings of the currently impersonated user.
    getRequest.Proxy = WebRequest.GetSystemWebProxy();

    getRequest.Credentials = new NetworkCredential("user@company.net", "password");
    using (var response = getRequest.GetResponse() as HttpWebResponse) {
        var sr = new StreamReader(response.GetResponseStream());
        HttpContext.Current.Response.Write(sr.ReadToEnd());
    }

.

HTTP Request only works in browser

3 Answers3