Why does HttpWorkerRequest fail during HttpRuntime.ProcessRequest after a .NET 2.0 to .NET 4.0 upgrade?

Question

I am upgrading our application, which has an internal webserver, from .NET 2.0 to .NET 4.0.

I am handling a request with an object HttpListenerWorkerRequest, that extends the HttpWorkerRequest class, and creates a request which GetRawUrl() returns a Url in the format of http://localhost:82/Default.aspx.

In .NET 2.0, sending this to HttpRuntime.ProcessRequest(httpListenerWorkerRequest) works without issue, however in .NET 4.0, I get a web page with the nothing but the text "Bad Request" on it.

Cracking open HttpRuntime, I can see that Bad Requests are thrown from ProcessRequestInternal(HttpWorkerRequest wr), a private method that tries to build an HttpContext.

I tried this myself:

try
{
    //what's going on?
    hcontext = new HttpContext(workerRequest);
}
catch(Exception e)
{
    //Debugging break point here
}

Pre-update (.NET 2.0), it builds fine, post-update (.NET 4.0), I get a System.ArgumentException stating that

The relative virtual path 'http:/localhost:82/Default.aspx' is not allowed here, thrown at

at System.Web.VirtualPath.Create(String virtualPath, VirtualPathOptions options)
   at System.Web.HttpRequest.get_ClientFilePath()
   at System.Web.Security.CookielessHelperClass.RemoveCookielessValuesFromPath()
   at System.Web.HttpContext.Init(HttpRequest request, HttpResponse response)
   at System.Web.HttpContext..ctor(HttpWorkerRequest wr)
   at Talcasoft.Web.Hosting.HttpWorkerThread.Run(Object request) in      
   C:\[OurLibrary].Web\Hosting\HttpWorkerThread.cs:line 51

What has changed in .NET to cause this, and what can I do to get around it?

EDIT I have just noticed that the disallowed http: is followed by a single slash, not a double, although the GetRawUrl() in the request certainly returns a double.

http://www.asp.net/whitepapers/aspnet4/breaking-changes I think you hit the XSS change — Lex Li, Apr 18 '13 at 03:22
@Lex Li, thanks for that, it certainly looks possible. I'll have to test that tomorrow, but let you know how it goes. — johnc, Apr 18 '13 at 06:49
@Lex Li, not the case, unfortunately, thanks for the suggestion though — johnc, Apr 19 '13 at 04:16
Can you show the entire exception? Set your debugger the break on all exceptions and disable "my code only" so that the debugger pauses on the first exception (the one that is being caught in ProcessRequestInternal). — usr, Apr 21 '13 at 22:25
@usr Amended the question to include it's a System.ArgumentException at VirtualPath.Create (I can see this is getting called within ProcessRequestInternal). Is that the information you were requesting? — johnc, Apr 22 '13 at 00:04
@Andreas I agree. I didn't write the server code, but the person who did has left. It works in 2.0, so I am loathe to touch it until I understand how it broke. Stepping through it in 2.0 shows that nothing appears to have changed in the WorkerRequest — johnc, Apr 22 '13 at 00:09
Did you make any progress yet? I'm interested in whether my suggestions works for you. — usr, Apr 22 '13 at 19:19
@usr We had to take our kid to the doctor today, do my mind has been on other things. I'll let you know tomorrow. Thanks for the input — johnc, Apr 23 '13 at 07:25
usr and @NSGaga You are both worthy of receiving the bounty points, however, unless you know of some way to split the points, I have to choose. I thank you both enormously, but have decided, as NSGaga supplied the actual line of code I used to fix the problem (and he/she could use the additional rep more than usr) I will mark NSGaga as the accepted answer IF you work your last two comments into the post — johnc, Apr 24 '13 at 06:46

score 7 · Accepted Answer · edited May 23 '17 at 12:04

I'm not a 100% certain that this is the 'exact answer' but looks pretty close to me - and there is some more to write...

There seems to be a breaking change of a sort inside the VirtualPath class - and it's substantiated with how they are checking for illegal characters. (btw. you can google the 'VirtualPath source' for what seems a .NET 4 version of it).

Inside VirtualPath.Create a check is invoked for 'illegal virtual path characters'.

It first goes into registry ("HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\ASP.NET", "VerificationCompatibility") - to see if the compatibility mode 'illegal chars' should be used.

Based on that - I'm guessing (I don't have a way of checking this right now) - that if you set the above registry value (int) to 1 - you should get your methods working the old way and w/o any additional effort. Note: a IIS (or host process) restart may be required as suggested in one of the links

And then based on that registry flag it used either of these two...

':', '?', '*', '\0' // .NET 4.0 - the default
'\0' // the 'compatibility' mode

That seems to actually describe your story quite well as your path with the 'port' designation is actually illegal per the new default.

FINAL EDIT / EXPLANATION:

(update based on comments and the solution that solved it)
This is my understanding of what's going on inside:

1) Solution for prior to .NET 4.0 was the VerificationCompatibility key (see above).

2) With .NET 4.0 internal handling and fixing of url paths is made more robust. And works fine in most cases. In short, all paths are fixed and normalized before entering the VirtualPath.Create - and your http://... becomes an expected absolute path /Default.aspx.

However when you supply the HttpWorkerRequest (instead of Request/Response etc.) - the raw Url is taken directly from the worker - and the responsibility for supplying the correct and normalized paths is down to your worker request. (this is still a bit iffy, and looks like a bug or bad handling internally).

To reproduce the issue:

internal class MyRequest : SimpleWorkerRequest
{
    public MyRequest() : base("", "", String.Empty, String.Empty, null) { }
    public override String GetRawUrl() { return "http://localhost:82/Default.aspx"; }
}
// ...
var wr = new MyRequest();
var context1 = new HttpContext(wr);

Gives the error The relative virtual path 'http:/localhost:82/Default.aspx' is not allowed here.

The FIX:

public override String GetRawUrl() 
{ return new Uri(url, UriKind.RelativeOrAbsolute).AbsolutePath; }

And some of the research on the subject based on that VerificationCompatibility keyword in the registry which seems to be the key to it.

Ampersand in URL filename = bad request

Configure IIS to accept URL with Special Characters...

For 32 vs 64 difference in Registry

And here is a similar thing from Microsoft - but what seems to be a 'hotfix' for '2.0', i.e. doesn't apply to you - but just attaching it as something official in this line.

FIX: "HTTP 400 - Bad request" error message in the .NET Framework 1.1
FIX: Error message when you try to visit an ASP.NET 2.0-based Web page: "HttpException (0x80004005): '/HandlerTest/WebForm1.aspx/a:b' is not a valid virtual path"
ASP.NET 2.0 x64 – You may get HTTP 400 Bad Request or Error as mentioned in KB 932552 or 826437

HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\ASP.NET

DWord Value Name: VerificationCompatibility Value Data: 1

It looks like the right answer, but still doesn't work. I'm going to drill down a bit further on this idea before I give up on it though. BTW, I'm using dotPeek to reflect on the code. Thanks for the pointers! — johnc, Apr 22 '13 at 03:16
I tried fast and it's not easy to reproduce (context, request). But if you use a custom `HttpWorkerRequest` and just override `GetRawUrl()` to return your url - it fails (and this being the culprit `VirtualPath.Create("...", VirtualPathOptions.AllowAbsolutePath)` - called internally and you have no 'say' there). The registry setting indeed doesn't seem to have any effect on that so I confirmed that. **I think it's down to your `HttpListenerWorkerRequest.GetRawUrl()` which needs to return the proper absolute path (w/o host:port)**. — NSGaga-mostly-inactive, Apr 22 '13 at 14:50
i.e. something like... `public override String GetRawUrl() { return new Uri(url, UriKind.RelativeOrAbsolute).AbsolutePath; }` in your `HttpListenerWorkerRequest` - should `fix` the problem. — NSGaga-mostly-inactive, Apr 22 '13 at 15:07
thanks for your help on this. I had to take the day off, so I'll have a look at your work tomorrow. — johnc, Apr 23 '13 at 07:26
the line public override String GetRawUrl() { return new Uri(url, UriKind.RelativeOrAbsolute).AbsolutePath; } worked perfectly! Thank you so much for your time on this! — johnc, Apr 24 '13 at 02:02
you're welcome John - glad it helped. I just made a final edit to incorporate what's in the comments here. I think that's pretty much the whole story. I don't want to make your life more complicated, just to wrap this up into a proper answer. Whatever you do is fine, Cheers. — NSGaga-mostly-inactive, Apr 24 '13 at 11:36

score 2 · Answer 2 · answered Apr 22 '13 at 09:37

Here is my take on it. It still contains some guesswork but I'll include a test that you can make to prove or disprove this hypothesis.

The stack trace shows ClientFilePath.get as the origin of the exception. It looks like this:

    if (this._clientFilePath == null)
    {
        string rawUrl = this.RawUrl;
        int index = rawUrl.IndexOf('?');
        if (index > -1)
        {
            rawUrl = rawUrl.Substring(0, index);
        }
        this._clientFilePath
     = VirtualPath.Create(rawUrl, VirtualPathOptions.AllowAbsolutePath); //here!
    }
    return this._clientFilePath;

It creates a VirtualPath and allows only absolute values. http://localhost:82/Default.aspx is a relative path because it does not start with a slash. It is not a URL in this context because it is not being interpreted as such.

So VirtualPath.Create understandably denies this path. I don't know why .NET 2.0 allowed this, but .NET 4.0 requires an absolute path and according to the code this is non-configurable.

Actually, I have never seen HttpRequest.RawUrl return a URL before. In my experience it should return an absolute path like /Default.aspx. If you want to set the host and port you have to find other means to do that.

So the fix is to not use http://localhost:82/Default.aspx but /Default.aspx. Would that work for you?

That was exactly the problem. Thanks so much for your time and help on this. Unfortunately as there are 2 of you supplying this answer, I'm a bit stumped as to who to accept as the bounty winner. — johnc, Apr 24 '13 at 02:06
I think you should accept the answer which explains the problem most clearly and which will be most useful for future visitors who have this exact problem. I'll leave that for you to judge. — usr, Apr 24 '13 at 10:58

Why does HttpWorkerRequest fail during HttpRuntime.ProcessRequest after a .NET 2.0 to .NET 4.0 upgrade?

2 Answers2

FINAL EDIT / EXPLANATION: