WebRequest Strange NotFound Error

Question

I have 2 different ASP.NET Core websites: admin and public. Both running on staging server and on local machine.

I send GET request to different pages to determine execution time of different pages and encountered problem with admin site: all urls on local instance and staging always returns 404 error:

An unhandled exception of type 'System.Net.WebException' occurred in System.dll

Additional information: The remote server returned an error: (404) Not Found.

Meanwhile, same requests in browser return html pages normally. Requests through HttpWebRequest to public site always also return 200 Status Code (OK). Code for request I took here.

I tried to add all headers and cookies from browser request, but it didn't help. Also tried to debug local instance and found that no exceptions thrown while request executed.

Any ideas?

404 means something requested in server is not there, like default website (index.html)... Try using a simple `WebClient w = new WebClient(); string s = w.DownloadString("http://myserver.com");` — , Aug 10 '16 at 12:06
Just out of curiosity, you've put a solid bounty on your own question with *One or more of the answers is exemplary and worthy of an additional bounty.*. There is just one answer, so it should be the `exemplary`, but you didn't accepted or even upvoted it. Strange. — Ivan Stoev, Jan 19 '17 at 12:12
Use fiddler to capture the communication for both the working and the failed attempt. Once this is done you can usually tell what needs to change. — Brian from state farm, Jan 19 '17 at 20:21
@Evgeny Levin though it's not a good practice, it is possible to configure a web application to return 404 error when authentication or authorization fails. In the link you provided no authentication is done. Maybe problem is that authentication fails but server returns a wrong response. — Omid, Jan 20 '17 at 19:24
My 5 cents - browser can send request via proxy; if so make sure your crafted requests goes trough the same proxy. — Ondrej Svejdar, Jan 23 '17 at 10:33
@EvgenyLevin, provide a [mcve], used to reproduce the issue and from which possible answers can be derived. Have you considered using HttpClient? What are your limitations re framework versions? I ask because the referenced code if from 3 years ago and there may be more modern ways to achieve what you want. — Nkosi, Jan 23 '17 at 12:16
Did you add UserAgent and Accept? Usually browser sends many headers, out of them, most important ones are User Agent and Accept. Try to investigate your server code, see what is it expecting explicitly. — Akash Kava, Jan 23 '17 at 18:51
I think you are using your code to test your site and returning 404 when you do it by code but by browser it's ok. How do you send the url (both browser and by code)? — Tito, Jan 25 '17 at 22:35
Can you share code of how are you sending GET requests to different pages? — Chetan, Jan 26 '17 at 01:11

score 6 · Answer 1 · edited May 23 '17 at 12:02

404 is way to generic. The code provided in answer in your link (https://stackoverflow.com/a/16642279/571203) does no error handling - this is brilliant example of how you can get to troubles when you blindly copy code from stackoverflow :)

Modified code with error handling should look like:

string urlAddress = "http://google.com/rrr";

var request = (HttpWebRequest)WebRequest.Create(urlAddress);
string data = null;
string errorData = null;
try
{
  using (var response = (HttpWebResponse)request.GetResponse())
  {
    data = ReadResponse(response);
  }
}
catch (WebException exception)
{
  using (var response = (HttpWebResponse)exception.Response)
  {
    errorData = ReadResponse(response);
  }
}

static string ReadResponse(HttpWebResponse response)
{
  if (response.CharacterSet == null)
  {
    using (var reader = new StreamReader(response.GetResponseStream()))
    {
      return reader.ReadToEnd();
    }
  }
  using (var reader = new StreamReader(response.GetResponseStream(), Encoding.GetEncoding(response.CharacterSet)))
  {
    return reader.ReadToEnd();
  }
}

So when there is an exception, you'll get not just the status code, but entire response from server in the errorData variable.

One thing to check is proxy - browser can use http proxy while your server client uses none.

WebRequest Strange NotFound Error

1 Answers1