HttpWebRequest.GetResponse() returning 404 Error

Question

I have some code that calls HttpWebRequest's GetResponse() method to retrieve HTML from a URL and return it to the calling method.

This has been working perfectly fine within my Development and QA environments but now that I have uploaded it to my UAT server, I keep getting the following error:

The remote server returned an error: (404) Not Found.

The main difference between Dev/QA and UAT is that UAT uses SSL/HTTPS based URLs whereas Dev/QA uses HTTP. I introduced the following line of code to help progress me a little futher:

ServicePointManager.ServerCertificateValidationCallback = new System.Net.Security.RemoteCertificateValidationCallback(AcceptAllCertifications);

where AcceptAllCertifications always returns true but I still get my 404 error.

I that people who previously had this error have been able to resolve the issue by merely ensuring the URI used for the HttpWebRequest doesn't have a slash at the end (see: Simple HttpWebRequest over SSL (https) gives 404 Not Found under C#) but this does not make a difference to me.

I have now tried what was suggested at this post (see: HttpWebResponse returns 404 error) where I render the exception on the page. This bypassed the yellow-warning screen and gives me a bit more informtion, including the URL it is trying to get a response from. However, when I copy and paste the URL into my browser, it works perfectly fine and renders the HTML on the page. I'm quite happy therefore that the correct URL is being used in the GetResponse call.

Has anyone got any ideas as to what may be causing me this grief? As said, it only seems to be a problem on my UAT server where I am using SSL.

Here is my code to assist:

public static string GetHtmlValues()
    {

        var webConfigParentUrlValue = new Uri(ConfigurationManager.AppSettings["ParentUrl"]);

        var destinationUrl = HttpContext.Current.Request.Url.AbsoluteUri;

        var path = "DestinationController" + "/" + "DestinationAction" + "?destinationUrl=" + destinationUrl;

        var redirect = new Uri(webConfigParentUrlValue, path).AbsoluteUri;
        ServicePointManager.ServerCertificateValidationCallback = new System.Net.Security.RemoteCertificateValidationCallback(AcceptAllCertifications);
        var request = (HttpWebRequest)WebRequest.Create(redirect);

//Ensures that if the user has already signed in to the application,
// their authorisation is carried on through to this new request
        AttachAuthorisedCookieIfExists(request);

        HttpWebResponse result;

        try
        {
            result = (HttpWebResponse)request.GetResponse();
        }
        catch (WebException ex)
        {
            result = ex.Response as HttpWebResponse;
        }

        String responseString;

        using (Stream stream = result.GetResponseStream())
        {
            StreamReader reader = new StreamReader(stream, Encoding.UTF8);
            responseString = reader.ReadToEnd();
        }

        return responseString;
    }

More details of the error as it is rendered on the page:

Answer 1

I ran into a similar situation, but with a different error message. My problem turned out to be that my UAT environment was Windows 2008 with .NET 4.5. In this environment the SSL handshake/detecting is performed differently than most web browsers. So I was seeing the URL render without error in a web browser but my application would generate an error. My error message included "The underlying connection was closed: An unexpected error occurred on a send". This might be your issue.

My solution was to force the protocol change. I detect the specific error, then I force a change in the security protocol of my application and try again.

This is the code I use:

catch (Exception ex)
{
    if(ex.Message.Contains("The underlying connection was closed: An unexpected error occurred on a send."))
    {
        ServicePointManager.SecurityProtocol = SecurityProtocolType.Ssl3;
        // retry the retrieval
    }
}

Answer 2

I finally found the solution to my problem...

The first clue to get me on the right track was the wrong physical path being displayed in the 404 error from IIS. It turns out that this incorrect physical path was mapped to another site in my IIS setup. This particular naturally had a binding also; port 443. As you may know, port 443 is the default port for https.

Now looking at my URL that I was trying to pass into the HTTPWebRequest.GetResponse() method, it looked something like this:

https://www.my-web-site.com

Taking this into account, when this application was hosted on IIS within the bounds of SSL, the error was occuring as follows:

1. Code enters the aforementioned method GetHtmlValues()
2. The code gets https://www.my-web-site.com from the web.config file
3. A response is requested from https://www.my-web-site.com
4. At this point, as no port has been specified and application is now out there on the open internet, it tries to get a response from https://www.my-web-site.com:443
5. The problem is, my application isn't hosted via IIS on port 443. A different application lives here. Subsequently, as the page can't be found on port 443, a 404 error is produced.

Now for the solution...

1. Looking in IIS, I found the port that my application sits on. Let's say port 16523.
2. Whereas previously in my web.config I had my key of ParentUrl decalred with a value of https://www.my-web-site.com, this is to be changed to http://www.my-web-site.com:16523

Note how the https has become http and the port number is specified at the end. Now when the application tries to get the response, it no longer uses the default ssl port as the correct one was specified.