5

I'm trying to execute the following command in PowerShell, but I have no idea how to escape the ampersand character which is part of the URL

  az rest `
    --method GET `
    --uri ("https://graph.microsoft.com/v1.0/groups?`$count=true&`$filter=startsWith(displayName,'some+filter+text')&`$select=id,displayName") `
    --headers 'Content-Type=application/json'

As the & character is used to start a new command, it breaks the url and want to execute the remainder.

Is there a way to tell powershell not to do that?

verbedr
  • 1,784
  • 1
  • 15
  • 18
  • 1
    Did you try to use the same character you used to escape the dollar signs? – Olaf Jan 24 '22 at 23:46
  • You could try to use the [stop-parsing](https://learn.microsoft.com/en-us/powershell/module/microsoft.powershell.core/about/about_parsing?view=powershell-7.2#the-stop-parsing-token) token `--%` – Olaf Jan 24 '22 at 23:48
  • I would just use single-quotes for that URI so it doesn't get expanded and remove all the backticks. Alternatively, as per Olaf comment, just add a backtick to the ampersand as well. – Sage Pourpre Jan 25 '22 at 00:05
  • 1
    @Olaf both suggestions didn't work. I thought the trick was to break the strings in parts and concat again but using the single quote for the & elements. Like ("https://graph.microsoft.com/v1.0/groups?`$count=true" + '"&"' + "`$filter=startsWith(displayName,'some+filter+text')" + '"&"' + "`$select=id,displayName"). But then it will add the double quotes, leaving them out it fails again. – verbedr Jan 25 '22 at 00:08
  • I actually suggested either to use the backtick to escape the amepersants as well like you did it with the dollar signs or to use the stop-parsing token. Did you read the help I linked in my second comment? – Olaf Jan 25 '22 at 00:14
  • @SagePourpre backticks will not work, and putting everything in single quotes neither. Typical message is then: '$filter' is not recognized as an internal or external command, operable program or batch file. '$select' is not recognized as an internal or external command, operable program or batch file. – verbedr Jan 25 '22 at 00:14

2 Answers2

10

Olaf's answer provides an effective solution; let me add an explanation:

The source of the problem is a confluence of two behaviors:

  • When calling external programs, PowerShell performs on-demand double-quoting of each argument solely based on whether a given argument value contains spaces - otherwise, the argument is passed unquoted - irrespective of whether or not the value was originally quoted in the PowerShell command (e.g., cmd /c echo ab, cmd /c echo 'ab', and cmd /c echo "ab" all result in unquoted ab getting passed as the last token on the command line PowerShell rebuilds behind the scenes to ultimately use for execution).

  • The Azure az CLI is implemented as a batch file (az.cmd) and when a batch file is called, it is cmd.exe that parses the arguments given; surprisingly - and arguably inappropriately - it parses them as if the command had been submitted from inside a cmd.exe session.

As a result, if an argument is passed from PowerShell to a batch file that (a) contains no spaces, yet (b) contains cmd.exe metacharacters such as &, the call breaks.

A simple demonstration, using a cmd /c echo call as a stand-in for a call to a batch file:

# !! Breaks, because PowerShell (justifiably) passes *unquoted* a&b
# !! when it rebuilds the command line to invoke behind the scenes.
PS> cmd /c echo 'a&b'
a
'b' is not recognized as an internal or external command,
operable program or batch file.

There are three workarounds:

  • Use embedded "..." quoting:
# OK, but with a CAVEAT: 
#   Works as of PowerShell 7.2, but arguably *shouldn't*, because
#   PowerShell should automatically *escape* the embedded " chars. as ""
PS> cmd /c echo '"a&b"'
"a&b"

# Ditto, using an *expandable* (interpolating) PowerShell string:
PS> cmd /c echo "`"$HOME & Family; can't put a `$ value on that.`""
"C:\Users\jdoe & Family; can't put a $ value on that." # e.g. 
# OK, but with a CAVEAT:
#  Requires "..." quoting, but doesn't recognize *PowerShell* variables,
#  also doesn't support single-quoting and line continuation.
PS> cmd /c echo --% "a&b"
 "a&b"
  • Call via cmd /c and pass a single string encompassing the batch-file call and all its arguments, (ultimately) using cmd.exe's syntax.
# OK (remember, cmd /c echo stands for a call to a batch file, such as az.cmd)
# Inside the single string passed to the outer cmd /c call,
# be sure to use "...", as that is the only quoting cmd.exe understands.
PS> cmd /c 'cmd /c echo "a&b"'
"a&b"

# Ditto, using an *expandable* (interpolating) PowerShell string:
PS> cmd /c "cmd /c echo `"$HOME & Family; can't put a `$ value on that.`""
"C:\Users\jdoe & Family; can't put a $ value on that." # e.g. 

Taking a step back:

Now, wouldn't it be nice if you didn't have to worry about all these things? Especially since you may not know or care if a given CLI - such as az - just so happens to be implemented as a batch file?

As a shell, PowerShell should do its best to relay arguments faithfully behind the scenes, and allow the caller to focus exclusively on satisfying only PowerShell's syntax rules:

  • Unfortunately, PowerShell has to date (PowerShell 7.2) generally done a very poor job in this regard, irrespective of cmd.exe's quirks - see this answer for a summary.

  • With respect to cmd.exe's (batch-file call) quirks, PowerShell could predictably compensate for them in a future version - but it looks like that isn't going to happen, unfortunately; see GitHub issue #15143.

mklement0
  • 382,024
  • 64
  • 607
  • 775
  • Thank you for this great explanation! A small side note, though. I was using the first workaround in your list. But this one is not working all the time, it does work most of the time. For some reason in the call of my question the double quoting ('"&"') keeps including the " in the final output. – verbedr Jan 25 '22 at 13:39
  • Glad to hear it was helpful, @verbedr. That's curious re the first workaround. Note that `cmd`'s `echo` always includes the double quotes it sees on the command line, so the workaround prints verbatim `"a&b"`. Are you saying that _additional_ `"` are present? Can you give a simple example, ideally with `cmd /c echo`, or does it only happen with `az.cmd`? – mklement0 Jan 25 '22 at 14:59
  • Also, @verbedr: With the exception of an extra leading space with the `--%` technique (which shouldn't make a difference), all `cmd /c echo ` calls should receive the same argument: verbatim `"a&b"` – mklement0 Jan 25 '22 at 16:12
  • you got my attention ... I worked by elimination/trail and error to get the command in the initial request working. >>> --uri ("https://graph.microsoft.com/v1.0/groups?`$count=true" + '"&"' + "`$filter=startsWith" + '"("' + "displayName,'Test'" + '")&"' + "`$select=id,displayName") <<< so the interesting thing is the interaction with other characters in the string. Using '(' without escaping this, the error message would be something like >>"&"$select was unexpected at this time.<<. Could this explain things? – verbedr Jan 25 '22 at 21:45
  • @verbedr, the workaround requires using embedded `"..."` quoting around the _entire_ string content, not substrings; if you want to apply it to a double-quoted PowerShell string, use `"\`"...\`""` – mklement0 Jan 25 '22 at 22:37
  • Perhaps not clear in the original question, but I needed to set some values (the filter value). Using the ' prevents $($var) expansion. So that's why I'm breaking up my string. But now I realize it would be better to turn things around. So this is the final result >>('"https://graph.microsoft.com/v1.0/groups?$count=true&$filter=startsWith(displayName, ''"' + $groupName + '"'')&$select=id,displayName"')<<. Thx for your suggestions – verbedr Jan 25 '22 at 23:18
  • @verbedr, the lack of formatting makes the code in your comment hard to read, but, generally speaking, I suggested `"\`"...\`""`, because it is an [expandable (double-quoted) string (`"..."`)](https://learn.microsoft.com/en-us/powershell/module/microsoft.powershell.core/about/about_Quoting_Rules#double-quoted-strings) in which you can embed variable references and subexpressions; escape any `$` characters inside it that you want to be passed through _as-is_ as `\`$` – mklement0 Jan 25 '22 at 23:35
  • 1
    @verbedr, I've added expandable-string examples to the first and third workaround (the second workaround, based on `--%`, doesn't support string interpolation / use of PowerShell variables in general). – mklement0 Jan 25 '22 at 23:48
1

I don't have access to an Azure tennant right now to test and I actually don't have experiences with the Azure CLI in general but I'd expect this to work:

az rest `
    --method GET `
    --uri 'https://graph.microsoft.com/v1.0/groups?$count=true&$filter=startsWith(displayName,some+filter+text)&$select=id,displayName' `
    --headers 'Content-Type=application/json'

or this:

az rest --method GET --headers "Content-Type=application/json" `
    --% --uri "https://graph.microsoft.com/v1.0/groups?$count=true&$filter=startsWith(displayName,some+filter+text)&$select=id,displayName"

I only added the backticks for better readability - you may remove them in your actual code.

Olaf
  • 4,690
  • 2
  • 15
  • 23
  • When removing the backtick (line continuations) and everything is on 1 line, it works. Or mvoe the stop-parsing to the last line like this. ➜ az rest ` >> --method GET ` >> --headers 'Content-Type=application/json'` >> --% --uri "https://graph.microsoft.com/v1.0/groups?$count=true&$filter=startsWith(displayName,'some+filter+totest')&$select=id,displayName" The thing is that the filter part needs to be passed along and only %environ% variables can be used. Also the stop parsing only works on windows env. But if you update your awnser I will marke it. – verbedr Jan 25 '22 at 00:29
  • so you got it working? – Olaf Jan 25 '22 at 00:31
  • 1
    Yes, thanks for the assist, it was driving me crazy. The stop parsing token interferes with the back tick line continuation token. But if you change order, it is manageable. – verbedr Jan 25 '22 at 00:47
  • I changed my answer accordingly. Thanks. – Olaf Jan 25 '22 at 00:52
  • Olaf, the `--%` solution works, but the first solution - perhaps surprisingly - does not, because PowerShell won't double-quote the `'https://...'` argument behind the scenes, because it doesn't contain spaces, which makes the _batch file_ `az.cmd` break with non-double-quoted arguments that contain `&` characters. – mklement0 Jan 25 '22 at 17:32