Powershell - Invoke-WebRequest to a URL with literal '/' (%2F) in it

Question

I have been trying to access a URL with a / character in it from powershell, using the following command (it's a query to a gitlab server to retrieve a project called "foo/bar"):

Invoke-WebRequest https://server.com/api/v3/projects/foo%2Fbar -Verbose

Now, the odd thing is that using the PowerShell ISE or Visual Studio, the request is OK. When using PowerShell itself, the URL is automatically un-escaped and the request fails. E.g.

In ISE/VS:

$> Invoke-WebRequest https://server.com/api/v3/projects/foo%2Fbar -Verbose
VERBOSE: GET https://server.com/api/v3/projects/foo%2Fbar with 0-byte payload
VERBOSE: received 19903-byte response of content type application/json

StatusCode        : 200
StatusDescription : OK
Content           : .... data ....

In Powershell:

$> Invoke-WebRequest https://server.com/api/v3/projects/foo%2Fbar -Verbose
VERBOSE: GET https://server.com/api/v3/projects/foo/bar with 0-byte payload
Invoke-WebRequest : {"error":"404 Not Found"}
At line:1 char:1
+ Invoke-WebRequest 'https://server.com/api/v3/projects/foo%2Fbar ...
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    + CategoryInfo          : InvalidOperation: (System.Net.HttpWebRequest:HttpWebRequest) [Invoke-WebRequest], WebException
    + FullyQualifiedErrorId : WebCmdletWebResponseException,Microsoft.PowerShell.Commands.InvokeWebRequestCommand

I have tried adding single and double quotes around the URL, but nothing is helping.

What could be the reason for this behaviour, and how do I make PS not un-escape the URL string?

Environment: Windows 7, also tested on Server 2012R2 with same results.

$> $PSVersionTable

Name                           Value
----                           -----
PSVersion                      4.0
WSManStackVersion              3.0
SerializationVersion           1.1.0.1
CLRVersion                     4.0.30319.42000
BuildVersion                   6.3.9600.16406
PSCompatibleVersions           {1.0, 2.0, 3.0, 4.0}
PSRemotingProtocolVersion      2.2

[Right here](https://github.com/PowerShell/PowerShell/blob/b043b444db58801ca98bd72f3242a55bb4b8234b/src/Microsoft.PowerShell.Commands.Utility/commands/utility/WebCmdlet/FullClr/WebRequestPSCmdlet.FullClr.cs#L47) the code calls `[System.Net.WebRequest]::CreateUri()` which [does not allow escaped slashes](http://stackoverflow.com/q/781205/478656) as a known problem, and is allegedly fixed in .Net 4.5. Two workarounds mentioned in that question, not sure if either is easily usable with PowerShell - what should the result of the request be? — TessellatingHeckler, Mar 31 '17 at 00:21
Thanks for the links. I'd seen those, I also wasn't quite sure how to apply them in a powershell context. The result of the request should be some json data. — nik, Apr 03 '17 at 02:04

score 13 · Accepted Answer · answered Mar 31 '17 at 08:17

13

Try the URL through this function

function fixuri($uri){
  $UnEscapeDotsAndSlashes = 0x2000000;
  $SimpleUserSyntax = 0x20000;

  $type = $uri.GetType();
  $fieldInfo = $type.GetField("m_Syntax", ([System.Reflection.BindingFlags]::Instance -bor [System.Reflection.BindingFlags]::NonPublic));

  $uriParser = $fieldInfo.GetValue($uri);
  $typeUriParser = $uriParser.GetType().BaseType;
$fieldInfo = $typeUriParser.GetField("m_Flags", ([System.Reflection.BindingFlags]::Instance -bor [System.Reflection.BindingFlags]::NonPublic -bor [System.Reflection.BindingFlags]::FlattenHierarchy));
$uriSyntaxFlags = $fieldInfo.GetValue($uriParser);

$uriSyntaxFlags = $uriSyntaxFlags -band (-bnot $UnEscapeDotsAndSlashes);
$uriSyntaxFlags = $uriSyntaxFlags -band (-bnot $SimpleUserSyntax);
$fieldInfo.SetValue($uriParser, $uriSyntaxFlags);
}

$uri = New-Object System.Uri -ArgumentList ("https://server.com/api/v3/projects/foo%2Fbar")
fixuri $uri

answered Mar 31 '17 at 08:17

Oleg SH

443
5
11

1

Just to note, this is essentially the same solution as in [various](http://stackoverflow.com/questions/25596564/percent-encoded-slash-is-decoded-before-the-request-dispatch) [.net](http://stackoverflow.com/questions/5774183/how-to-make-system-uri-not-to-unescape-2f-slash-in-path) [solutions](http://stackoverflow.com/questions/7307833/creating-an-uri-in-net-automatically-urldecodes-all-parameters-from-passed-stri), although I had not seen a powershell implementation of it until this. – nik Apr 03 '17 at 02:13
Lol, an utterly ridiculous problem and a fix to match. We totally deserve this. – Daibhi O Domhnaill May 01 '19 at 13:46
Absolute waste of my time to have to stumble upon a hackabout like this.. way to go System.Uri! – user2864740 Jul 09 '19 at 21:36
And it seems that through recent updates of PowerShell, this doesn't work any more. I'm on 6.2 and `$fileInfo` is null, so the whole function fails. I have not yet found any way to get it to work; ditching PowerShell! – dariok Oct 23 '19 at 09:20

user2864740 · Answer 2 · 2019-07-09T22:05:56.117

Here is an alternate port of https://stackoverflow.com/a/784937/2864740 - it accepts a string and returns a new URI.

function CreateUriWithoutIncorrectSlashEncoding {
    param(
        [Parameter(Mandatory)][string]$uri
    )

    $newUri = New-Object System.Uri $uri

    [void]$newUri.PathAndQuery # need to access PathAndQuery (presumably modifies internal state)
    $flagsFieldInfo = $newUri.GetType().GetField("m_Flags", [System.Reflection.BindingFlags]::Instance -bor [System.Reflection.BindingFlags]::NonPublic)
    $flags = $flagsFieldInfo.GetValue($newUri)
    $flags = $flags -band (-bnot 0x30) # remove Flags.PathNotCanonical|Flags.QueryNotCanonical (private enum)
    $flagsFieldInfo.SetValue($newUri, $flags)

    $newUri
}

Usage:

$uri = CreateUriWithoutIncorrectSlashEncoding "https://server.com/api/v3/projects/foo%2Fbar"

I really like this solution. For those that read it and are confused, you would then use this in the Uri parameter of Invoke-RestMethod/Invoke-WebRequest `Invoke-RestMethod -Uri $uri` — omrsafetyo, Sep 28 '21 at 19:41

score 2 · Answer 3 · answered Jul 18 '17 at 06:47

I have encountered similar issue in PowerShell 5.1. My purpose was to get a single project by Git Lab Web API. As Web API described:

Get single project
Get a specific project, identified by project ID or NAMESPACE/PROJECT_NAME , which is owned by the authentication user. If using namespaced projects call make sure that the NAMESPACE/PROJECT_NAME is URL-encoded, eg. /api/v3/projects/diaspora%2Fdiaspora (where / is represented by %2F).

What different with nik was that my Invoke-WebRequest call was successful by directly invoke but failed inside a Job. Here's the code:

Start-Job -ScriptBlock {
    try{
        Invoke-WebRequest https://server.com/api/v3/projects/foo%2Fbar -verbose
    } catch {
        Write-Output $_.Exception
    }
}

To Get the output inside a Job. Run command:

> Get-Job | Receive-Job -Keep

And exception as below:

VERBOSE: GET https://server.com/api/v3/projects/foo/bar with 0-byte payload
System.Net.WebException: The remote server returned an error: (404) Not Found.
  at Microsoft.PowerShell.Commands.WebRequestPSCmdlet.GetResponse(WebRequest request)
  at Microsoft.PowerShell.Commands.WebRequestPSCmdlet.ProcessRecord()

And thanks Oleg SH's answer. My problem was solved. But I think there might be a bug in the Start-Job cmdlet

Environment: Windows 7

>$PSVersionTable

Name                           Value
----                           -----
PSVersion                      5.1.14409.1005
PSEdition                      Desktop
PSCompatibleVersions           {1.0, 2.0, 3.0, 4.0...}
BuildVersion                   10.0.14409.1005
CLRVersion                     4.0.30319.42000
WSManStackVersion              3.0
PSRemotingProtocolVersion      2.3
SerializationVersion           1.1.0.1

Powershell - Invoke-WebRequest to a URL with literal '/' (%2F) in it

3 Answers3

Linked