Prevent unescaping url in outbound request

Question

If I do this in .Net Core 3.1:

await new HttpClient().GetAsync("http://test.com/page?parameter=%2D%2E%5F%2E%2D");

then this happens:

GET http://test.com/page?parameter=-._.- HTTP/1.1

but this is what I want:

GET http://test.com/page?parameter=%2D%2E%5F%2E%2D HTTP/1.1

The background is that I get a signed Url from a third party and I need to use the url as it is, non-unescaped. I manage to find the resource with the unescaped url, but the signature check fails on the other end because the url they see in the request is not the url that was signed.

I can paste the url into any browser and get the resource, but the signature check fails when I do it programatically in .Net Core 3.1.

The unescaping is supposed to happen according to documentation on the Uri Class:

Escaped characters (also known as percent-encoded octets) that don't have a reserved purpose are decoded (also known as being unescaped). These unreserved characters include uppercase and lowercase letters (%41-%5A and %61-%7A), decimal digits (%30-%39), hyphen (%2D), period (%2E), underscore (%5F), and tilde (%7E).

I have tried solutions listed in these questions:

GETting a URL with an url-encoded slash. But the schemeSetting seems not to work for .Net Core 3.1 and and neither does the workaround ForceCanonicalPathAndQuery.
How to make System.Uri not to unescape %2f (slash) in path?. Again schemeSetting seems not to work for .Net Core 3.1, and neither does the workaround LeaveDotsAndSlashesEscaped.

So, does anyone know how I can use the signed url as is, non-unescaped, on .Net Core 3.1?

`http://test.com/page?parameter=-._.-` is a valid url, no need to encode — André Sanson, Jan 21 '20 at 18:53
Yes, but my url is `http://test.com/page?parameter=%2D%2E%5F%2E%2D` and the problem is that it is being decoded. I want to use it undecoded, non-unescaped, raw and as it is — Ulf Kristiansen, Jan 23 '20 at 11:07

Ulf Kristiansen · Answer 1 · 2022-04-09T10:16:51.560

So after fiddling around a bit I came up with this:

private static Uri CreateNonUnescapedUri(string url)
{
    // Initiate Uri as e.g "http://test.com" so internal flags will indicate that the url does not include characters that needs unescaping
    int offset = url.IndexOf("://");
    offset = url.IndexOf('/', offset + 4);
    var uri = new Uri(url.Substring(0, offset));
    
    // Then replace internal field with complete url that would otherwise be unescaped
    typeof(Uri).GetField("_string", BindingFlags.Instance | BindingFlags.NonPublic).SetValue(uri, url);
    return uri;
}

I tested it on 300 signed url's.

Offcourse changing the internal state of the Uri which is 5600 lines of pure madness is bound to fail in the future, but I need this working by monday and this is what I've got. Let me know if anyone has a real solution.

Edit April 2022:

In .Net 6 there is a new constructor that will keep the original url as is, using UriCreationOptions:

 var uri = new Uri("http://test.com/page?parameter=%2D%2E%5F%2E%2D",
           new UriCreationOptions { DangerousDisablePathAndQueryCanonicalization = true });

I have no idea whats supposedly dangerous about it though.

For .Net Core 3.1 I'm still using the hack above, I never did find a better solution for it.

Prevent unescaping url in outbound request

1 Answers1