Internal Server Error (500) with HttpWebRequest

Question

I'm trying to connect to a remote server (from our customer) via HttpWebRequest.GetResponse but at the login page i'm always getting an exception:

"The remote server returned an error: (500) Internal Server Error."

In the response stream i see that i was redirected to the error-page.

The code worked so far(HTTP status 200) with the old ASP-website but they'll migrate to ASP.NET soon. There's no error if i browse to the URL manually.

This is the (reduced) code:

Try
    Dim URL As String = "https://Domain.com/Modules/LoginPage.aspx"
    ' the old URL was similar: https://www.Domain.com/login.asp '  
    Dim req As System.Net.HttpWebRequest = CType(System.Net.WebRequest.Create(URL), System.Net.HttpWebRequest)
    req.Headers.Clear()
    req.AllowAutoRedirect = True
    req.Timeout = 1000 * 30
    req.PreAuthenticate = True
    req.UserAgent = "Mozilla/4.0 (compatible; MSIE 5.01; Windows NT 5.0)"
    req.Accept = "*/*"
    req.CookieContainer = New System.Net.CookieContainer
    ' following will result in an HTTP 500 Error
    Dim response As System.Net.HttpWebResponse = CType(req.GetResponse(), System.Net.HttpWebResponse)
Catch ex As System.Net.WebException
    ' ex.Response.ResponseUri shows the redirected ErrorPage.aspx
End Try

Although here are many similar questions, i haven't yet found one that would help me.

Q: Do you have any hints or ideas that could help me to find the reason? Is it possible that they somehow prevent that i can connect programmatically?

I've also used Fiddler2 and ieHTTPHeaders to analyze if there are any difference between the old and the new site, but i must admit that i'm not sure what must be given attention. The headers seem to be similar:

old "working" ASP site:

GET /login.asp HTTP/1.1
Accept: */*
Accept-Language: de
User-Agent: Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 5.1; Trident/4.0; DWE40; .NET CLR 1.1.4322; .NET CLR 3.0.04506.30; .NET CLR 2.0.50727; .NET CLR 3.0.04506.648; .NET CLR 3.0.4506.2152; .NET CLR 3.5.30729; OfficeLiveConnector.1.4; OfficeLivePatch.1.3; InfoPath.3; .NET4.0C; .NET4.0E)
Accept-Encoding: gzip, deflate
Host: www.domain.blah.com
Connection: Keep-Alive
Cache-Control: no-cache
Cookie: ASPSESSIONIDCSSTADCS=CJDLHKLANAELOKNOJOPCLCDG

HTTP/1.1 200 OK
Date: Wed, 20 Jun 2012 07:36:11 GMT
Server: Microsoft-IIS/6.0
X-Powered-By: ASP.NET
Content-Length: 3744
Content-Type: text/html
Cache-control: private

GET /StyleSheets/IE/FormStyle.css HTTP/1.1
Accept: */*
Referer: https://www.domain.blah.com/login.asp
Accept-Language: de
User-Agent: Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 5.1; Trident/4.0; DWE40; .NET CLR 1.1.4322; .NET CLR 3.0.04506.30; .NET CLR 2.0.50727; .NET CLR 3.0.04506.648; .NET CLR 3.0.4506.2152; .NET CLR 3.5.30729; OfficeLiveConnector.1.4; OfficeLivePatch.1.3; InfoPath.3; .NET4.0C; .NET4.0E)
Accept-Encoding: gzip, deflate
Host: www.domain.blah.com
Connection: Keep-Alive
Cache-Control: no-cache
Cookie: ASPSESSIONIDCSSTADCS=CJDLHKLANAELOKNOJOPCLCDG

HTTP/1.1 200 OK
Content-Length: 1116
Content-Type: text/css
Last-Modified: Thu, 11 Jul 2002 16:05:52 GMT
Accept-Ranges: bytes
ETag: "03060d4f428c21:31d"
Server: Microsoft-IIS/6.0
X-Powered-By: ASP.NET
Date: Wed, 20 Jun 2012 07:36:11 GMT

GET /JScripts/common.js HTTP/1.1
Accept: */*
Referer: https://www.domain.blah.com/login.asp
Accept-Language: de
User-Agent: Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 5.1; Trident/4.0; DWE40; .NET CLR 1.1.4322; .NET CLR 3.0.04506.30; .NET CLR 2.0.50727; .NET CLR 3.0.04506.648; .NET CLR 3.0.4506.2152; .NET CLR 3.5.30729; OfficeLiveConnector.1.4; OfficeLivePatch.1.3; InfoPath.3; .NET4.0C; .NET4.0E)
Accept-Encoding: gzip, deflate
Host: www.domain.blah.com
Connection: Keep-Alive
Cache-Control: no-cache
Cookie: ASPSESSIONIDCSSTADCS=CJDLHKLANAELOKNOJOPCLCDG

HTTP/1.1 200 OK
Content-Length: 16068
Content-Type: application/x-javascript
Last-Modified: Mon, 24 Jan 2011 10:37:53 GMT
Accept-Ranges: bytes
ETag: "8d4371c1b2bbcb1:31d"
Server: Microsoft-IIS/6.0
X-Powered-By: ASP.NET
Date: Wed, 20 Jun 2012 07:36:11 GMT

new ASP.NET site (HTTP 500)

GET /Modules/Authorization/LoginPage.aspx HTTP/1.1
Accept: image/gif, image/jpeg, image/pjpeg, image/pjpeg, application/x-shockwave-flash, application/xaml+xml, application/vnd.ms-xpsdocument, application/x-ms-xbap, application/x-ms-application, application/x-silverlight, application/vnd.ms-excel, application/vnd.ms-powerpoint, application/msword, */*
Accept-Language: de
User-Agent: Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 5.1; Trident/4.0; DWE40; .NET CLR 1.1.4322; .NET CLR 3.0.04506.30; .NET CLR 2.0.50727; .NET CLR 3.0.04506.648; .NET CLR 3.0.4506.2152; .NET CLR 3.5.30729; OfficeLiveConnector.1.4; OfficeLivePatch.1.3; InfoPath.3; .NET4.0C; .NET4.0E)
Accept-Encoding: gzip, deflate
Host: mo.domain.blah.com
Connection: Keep-Alive

HTTP/1.1 200 OK
Cache-Control: private
Content-Type: text/html; charset=utf-8
Server: Microsoft-IIS/7.5
Set-Cookie: ASP.NET_SessionId=if3igvajwackerbutp2ad21i; path=/; secure; HttpOnly
X-AspNet-Version: 4.0.30319
X-Powered-By: ASP.NET
Date: Wed, 20 Jun 2012 07:36:55 GMT
Content-Length: 19363

GET /JQuery/css/smoothness/jquery-ui-1.8.18.custom.css HTTP/1.1
Accept: */*
Referer: https://mo.domain.blah.com/Modules/Authorization/LoginPage.aspx
Accept-Language: de
User-Agent: Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 5.1; Trident/4.0; DWE40; .NET CLR 1.1.4322; .NET CLR 3.0.04506.30; .NET CLR 2.0.50727; .NET CLR 3.0.04506.648; .NET CLR 3.0.4506.2152; .NET CLR 3.5.30729; OfficeLiveConnector.1.4; OfficeLivePatch.1.3; InfoPath.3; .NET4.0C; .NET4.0E)
Accept-Encoding: gzip, deflate
If-Modified-Since: Wed, 29 Feb 2012 21:52:34 GMT
If-None-Match: "07d65712cf7cc1:0"
Host: mo.domain.blah.com
Connection: Keep-Alive
Cookie: ASP.NET_SessionId=if3igvajwackerbutp2ad21i

HTTP/1.1 304 Not Modified
Last-Modified: Wed, 29 Feb 2012 21:52:34 GMT
Accept-Ranges: bytes
ETag: "07d65712cf7cc1:0"
Server: Microsoft-IIS/7.5
X-Powered-By: ASP.NET
Date: Wed, 20 Jun 2012 07:36:55 GMT

GET /App_Themes/domain/Site.css HTTP/1.1
Accept: */*
Referer: https://mo.domain.blah.com/Modules/Authorization/LoginPage.aspx
Accept-Language: de
User-Agent: Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 5.1; Trident/4.0; DWE40; .NET CLR 1.1.4322; .NET CLR 3.0.04506.30; .NET CLR 2.0.50727; .NET CLR 3.0.04506.648; .NET CLR 3.0.4506.2152; .NET CLR 3.5.30729; OfficeLiveConnector.1.4; OfficeLivePatch.1.3; InfoPath.3; .NET4.0C; .NET4.0E)
Accept-Encoding: gzip, deflate
If-Modified-Since: Wed, 30 May 2012 12:03:02 GMT
If-None-Match: "0dfa1295c3ecd1:0"
Host: mo.domain.blah.com
Connection: Keep-Alive
Cookie: ASP.NET_SessionId=if3igvajwackerbutp2ad21i

HTTP/1.1 304 Not Modified
Last-Modified: Wed, 30 May 2012 12:03:02 GMT
Accept-Ranges: bytes
ETag: "0dfa1295c3ecd1:0"
Server: Microsoft-IIS/7.5
X-Powered-By: ASP.NET
Date: Wed, 20 Jun 2012 07:36:55 GMT

GET /CommonControls/Common.js HTTP/1.1
Accept: */*
Referer: https://mo.domain.blah.com/Modules/Authorization/LoginPage.aspx
Accept-Language: de
User-Agent: Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 5.1; Trident/4.0; DWE40; .NET CLR 1.1.4322; .NET CLR 3.0.04506.30; .NET CLR 2.0.50727; .NET CLR 3.0.04506.648; .NET CLR 3.0.4506.2152; .NET CLR 3.5.30729; OfficeLiveConnector.1.4; OfficeLivePatch.1.3; InfoPath.3; .NET4.0C; .NET4.0E)
Accept-Encoding: gzip, deflate
If-Modified-Since: Wed, 18 Apr 2012 12:19:12 GMT
If-None-Match: "0a072765d1dcd1:0"
Host: mo.domain.blah.com
Connection: Keep-Alive
Cookie: ASP.NET_SessionId=if3igvajwackerbutp2ad21i

HTTP/1.1 304 Not Modified
Last-Modified: Wed, 18 Apr 2012 12:19:12 GMT
Accept-Ranges: bytes
ETag: "0a072765d1dcd1:0"
Server: Microsoft-IIS/7.5
X-Powered-By: ASP.NET
Date: Wed, 20 Jun 2012 07:36:55 GMT

GET /App_Themes/domain/images/Gray-white-gradient-vertical.gif HTTP/1.1
Accept: */*
Referer: https://mo.domain.blah.com/Modules/Authorization/LoginPage.aspx
Accept-Language: de
User-Agent: Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 5.1; Trident/4.0; DWE40; .NET CLR 1.1.4322; .NET CLR 3.0.04506.30; .NET CLR 2.0.50727; .NET CLR 3.0.04506.648; .NET CLR 3.0.4506.2152; .NET CLR 3.5.30729; OfficeLiveConnector.1.4; OfficeLivePatch.1.3; InfoPath.3; .NET4.0C; .NET4.0E)
Accept-Encoding: gzip, deflate
If-Modified-Since: Fri, 27 Jan 2012 13:47:28 GMT
If-None-Match: "0983c35fadccc1:0"
Host: mo.domain.blah.com
Connection: Keep-Alive
Cookie: ASP.NET_SessionId=if3igvajwackerbutp2ad21i

HTTP/1.1 200 OK
Cache-Control: public
Content-Type: text/javascript
Content-Encoding: gzip
Expires: Thu, 20 Jun 2013 07:36:55 GMT
Last-Modified: Wed, 20 Jun 2012 07:36:55 GMT
Server: Microsoft-IIS/7.5
X-AspNet-Version: 4.0.30319
X-Powered-By: ASP.NET
Date: Wed, 20 Jun 2012 07:36:55 GMT
Content-Length: 1929

HTTP/1.1 304 Not Modified
Last-Modified: Fri, 27 Jan 2012 13:47:28 GMT
Accept-Ranges: bytes
ETag: "0983c35fadccc1:0"
Server: Microsoft-IIS/7.5
X-Powered-By: ASP.NET
Date: Wed, 20 Jun 2012 07:36:55 GMT

GET /App_Themes/domain/images/customername-Logo_148x38.png HTTP/1.1
Accept: */*
Referer: https://mo.domain.blah.com/Modules/Authorization/LoginPage.aspx
Accept-Language: de
User-Agent: Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 5.1; Trident/4.0; DWE40; .NET CLR 1.1.4322; .NET CLR 3.0.04506.30; .NET CLR 2.0.50727; .NET CLR 3.0.04506.648; .NET CLR 3.0.4506.2152; .NET CLR 3.5.30729; OfficeLiveConnector.1.4; OfficeLivePatch.1.3; InfoPath.3; .NET4.0C; .NET4.0E)
Accept-Encoding: gzip, deflate
If-Modified-Since: Sun, 26 Feb 2012 10:13:52 GMT
If-None-Match: "0b0b2566ff4cc1:0"
Host: mo.domain.blah.com
Connection: Keep-Alive
Cookie: ASP.NET_SessionId=if3igvajwackerbutp2ad21i

HTTP/1.1 304 Not Modified
Last-Modified: Sun, 26 Feb 2012 10:13:52 GMT
Accept-Ranges: bytes
ETag: "0b0b2566ff4cc1:0"
Server: Microsoft-IIS/7.5
X-Powered-By: ASP.NET
Date: Wed, 20 Jun 2012 07:36:56 GMT

HTTP/1.1 304 Not Modified
Last-Modified: Fri, 24 Feb 2012 11:19:44 GMT
Accept-Ranges: bytes
ETag: "0887235e6f2cc1:0"
Server: Microsoft-IIS/7.5
X-Powered-By: ASP.NET
Date: Wed, 20 Jun 2012 07:36:56 GMT

Edit: As requested, here are the headers from the HttpWebRequest monitored by Fiddler. The strange thing is - there is no HTTP 500 error, why do i get the exception anyway?

CONNECT mo.domain.blah.com:443 HTTP/1.1
Host: mo.domain.blah.com

HTTP/1.1 200 Connection Established
FiddlerGateway: Direct
StartTime: 10:20:32.948
Connection: close

This is a CONNECT tunnel, through which encrypted HTTPS traffic flows.
To view the encrypted sessions inside this tunnel, enable the Tools > Fiddler Options > HTTPS > Decrypt HTTPS traffic option.

A SSLv3-compatible ServerHello handshake was found. Fiddler extracted the parameters below.

No Proxy-Authorization Header is present.
No Authorization Header is present.

This request did not send any cookie data.

If i disable redirection, i get HTTP 302 with "Object moved":

req.AllowAutoRedirect = False

result:

HTTP/1.1 302 Found
Cache-Control: private
Content-Type: text/html; charset=utf-8
Location: https://mo.domain.blah.com/ErrorPage.aspx?aspxerrorpath=/Modules/Authorization/LoginPage.aspx
Server: Microsoft-IIS/7.5
X-AspNet-Version: 4.0.30319
X-Powered-By: ASP.NET
Date: Wed, 20 Jun 2012 10:56:25 GMT
Content-Length: 216

<html><head><title>Object moved</title></head><body>
<h2>Object moved to <a href="https://mo.domain.blah.com/ErrorPage.aspx?aspxerrorpath=/Modules/Authorization/LoginPage.aspx">here</a>.</h2>
</body></html>

After i've enabled the decrypt HTTPS traffic option in fiddler, i could "see" the default ASP.NET error-page:

HTTP/1.1 500 Internal Server Error
Cache-Control: private
Content-Type: text/html; charset=utf-8
Server: Microsoft-IIS/7.5
X-AspNet-Version: 4.0.30319
X-Powered-By: ASP.NET
Date: Wed, 20 Jun 2012 10:49:29 GMT
Content-Length: 3030

<html>
    <head>
        <title>Runtime Error</title>
        <style>
         body {font-family:"Verdana";font-weight:normal;font-size: .7em;color:black;} 
         p {font-family:"Verdana";font-weight:normal;color:black;margin-top: -5px}
         b {font-family:"Verdana";font-weight:bold;color:black;margin-top: -5px}
         H1 { font-family:"Verdana";font-weight:normal;font-size:18pt;color:red }
         H2 { font-family:"Verdana";font-weight:normal;font-size:14pt;color:maroon }
         pre {font-family:"Lucida Console";font-size: .9em}
         .marker {font-weight: bold; color: black;text-decoration: none;}
         .version {color: gray;}
         .error {margin-bottom: 10px;}
         .expandable { text-decoration:underline; font-weight:bold; color:navy; cursor:hand; }
        </style>
    </head>

    <body bgcolor="white">

            <span><H1>Server Error in '/' Application.<hr width=100% size=1 color=silver></H1>

            <h2> <i>Runtime Error</i> </h2></span>

            <font face="Arial, Helvetica, Geneva, SunSans-Regular, sans-serif ">

            <b> Description: </b>An application error occurred on the server. The current custom error settings for this application prevent the details of the application error from being viewed remotely (for security reasons). It could, however, be viewed by browsers running on the local server machine.
            <br><br>

            <b>Details:</b> To enable the details of this specific error message to be viewable on remote machines, please create a &lt;customErrors&gt; tag within a &quot;web.config&quot; configuration file located in the root directory of the current web application. This &lt;customErrors&gt; tag should then have its &quot;mode&quot; attribute set to &quot;Off&quot;.<br><br>

            <table width=100% bgcolor="#ffffcc">
               <tr>
                  <td>
                      <code><pre>

&lt;!-- Web.Config Configuration File --&gt;

&lt;configuration&gt;
    &lt;system.web&gt;
        &lt;customErrors mode=&quot;Off&quot;/&gt;
    &lt;/system.web&gt;
&lt;/configuration&gt;</pre></code>

                  </td>
               </tr>
            </table>

            <br>

            <b>Notes:</b> The current error page you are seeing can be replaced by a custom error page by modifying the &quot;defaultRedirect&quot; attribute of the application&#39;s &lt;customErrors&gt; configuration tag to point to a custom error page URL.<br><br>

            <table width=100% bgcolor="#ffffcc">
               <tr>
                  <td>
                      <code><pre>

&lt;!-- Web.Config Configuration File --&gt;

&lt;configuration&gt;
    &lt;system.web&gt;
        &lt;customErrors mode=&quot;RemoteOnly&quot; defaultRedirect=&quot;mycustompage.htm&quot;/&gt;
    &lt;/system.web&gt;
&lt;/configuration&gt;</pre></code>

                  </td>
               </tr>
            </table>

            <br>

    </body>
</html>

Any kind of help is much appreciated since i'm running out of ideas.

I don't see any HTTP 500 response in your last section of script, just lots of 200 and 304 lol — Alvin Wong, Jun 20 '12 at 07:49
silly question, have you tried to go to the page via a browser? maybe there is an error on the server? — Qpirate, Jun 20 '12 at 07:50
@AlvinWong: These are the headers recorded from ieHTTPHeaders when i browse to the urls manually (that works without error as mentioned). My problem is that i cannot connect programmatically via [`HttpWebRequest`](http://msdn.microsoft.com/en-us/library/system.net.httpwebrequest.aspx). — Tim Schmelter, Jun 20 '12 at 07:54
@TimSchmelter so perhaps show us the request and response you get using VB.NET — Alvin Wong, Jun 20 '12 at 07:58
@AlvinWong: I didn't know that [i can](http://stackoverflow.com/questions/1470634/get-http-requests-and-responses-made-using-httpwebrequest-httpwebresponse-to-sho). I'll try to monitor that, but that'll take some time. — Tim Schmelter, Jun 20 '12 at 08:01
Perhaps you can alternatively see if the server has anything logged for your 500 error — Alvin Wong, Jun 20 '12 at 08:06
@AlvinWong: The server is beyond my control (it's our customer but ... ) — Tim Schmelter, Jun 20 '12 at 08:09
@AlvinWong: Edited my answer and added fiddler headers from programmatic request. Am i missing something? — Tim Schmelter, Jun 20 '12 at 08:26
The only changed thing is the server, so that *is* probably the problem — Alvin Wong, Jun 20 '12 at 08:32
As mentioned in my question, i'm getting redirected to the error page. Is it possible that the HTTP 500 is thrown there and not on the login page(the site is still beta)? @AlvinWong: The server is the same(where do you see that since i've obscured it?). But i get still the exception in the code, even if there's no http 500 in the header. — Tim Schmelter, Jun 20 '12 at 08:32
It says `To view the encrypted sessions inside this tunnel, enable the Tools > Fiddler Options > HTTPS > Decrypt HTTPS traffic option.` have you tried this? — Alvin Wong, Jun 20 '12 at 08:44
@AlvinWong: Good advice. Edited my answer and added the result. Now i'm able to "see" the error page, but that doesn't help me, does it? — Tim Schmelter, Jun 20 '12 at 11:11

score 1 · Answer 1 · answered Jun 20 '12 at 11:19

1

You are having a runtime error in your ASP script. Fix it and it'll be fine.

Set this to enable detailed error message (according to your error page)

<!-- Web.Config Configuration File -->

<configuration>
    <system.web>
        <customErrors mode="Off"/>
    </system.web>
</configuration>

answered Jun 20 '12 at 11:19

Alvin Wong

12,210
5
51
77

Thanks. But the server is beyond my control. So i neither can change the web.config nor have a look at the windows logs. The page can be browsed manually without a problem, why i'm being redirected from the login- to the error page when i connect via `HttpWebRequest`? – Tim Schmelter Jun 20 '12 at 11:24
No difference. No error when i'm using the browser, internal server error when i use HttpWebRequest with the same URL(Login.aspx). But +1 anyway for your former afford, thanks. – Tim Schmelter Jun 20 '12 at 11:35
Yes, probably, but no chance. It's developed by HP for our customer. The site is still in beta and will go live next month, but i doubt that there's a problem on the login page which only happens when connecting programmatically. Is that even possible? – Tim Schmelter Jun 20 '12 at 11:39

Internal Server Error (500) with HttpWebRequest

1 Answers1