1

Performing URLdecoding in CGI hybrid script by powershell oneliner:

echo %C4%9B%C5%A1%C4%8D%C5%99%C5%BE%C3%BD%C3%A1%C3%AD%C3%A9%C5%AF%C3%BA| powershell.exe "Add-Type -AssemblyName System.Web;[System.Web.HttpUtility]::UrlDecode($Input) | Write-Host"

execution time of this oneliner is between 2-3 seconds on virtual machine. Is it because of .NET object is employed? Is there any way to decrease execution time? Have also some C written, lightning fast urldecode.exe utility, but unfortunatelly it does not eat STDIN.

user2956477
  • 1,208
  • 9
  • 17

1 Answers1

4

A note if you're passing the input data as a string literal, as in the sample call in the question (rather than as output from a command):

  • If you're calling from an interactive cmd.exe session, %C4%9B%C5%A1%C4%8D%C5%99%C5%BE%C3%BD%C3%A1%C3%AD%C3%A9%C5%AF%C3%BA works as-is - unless any of the tokens between paired % instances happen to be the name of an existing environment variable (which seems unlikely).

  • From a batch file, you'll have to escape the % chars. by doubling them - see this answer; which you can obtain by applying the following PowerShell operation on the original string:

    '%C4%9B%C5%A1%C4%8D%C5%99%C5%BE%C3%BD%C3%A1%C3%AD%C3%A9%C5%AF%C3%BA' -replace '%', '%%'
    

Is it because of .NET object is employed?

  • Yes, powershell.exe, as a .NET-based application requires starting the latter's runtime (CLR), which is nontrivial in terms of performance.

  • Additionally, powershell.exe by default loads the initialization files listed in its $PROFILE variable, which can take additional time.

    • Pass the -NoProfile CLI option to suppress that.

Have also some C written, lightning fast urldecode.exe utility, but unfortunately it does not eat STDIN.

If so, pass the data as an argument, if feasible; e.g.:

urldecode.exe "%C4%9B%C5%A1%C4%8D%C5%99%C5%BE%C3%BD%C3%A1%C3%AD%C3%A9%C5%AF%C3%BA"

If the data comes from another command's output, you can use for /f to capture in a variable first, and then pass the latter.


If you do need to call powershell.exe, PowerShell's CLI, after all:

There's not much you can do in terms of optimizing performance:

  • Add -NoProfile, as suggested.
  • Pass the input data as an argument.
  • Avoid unnecessary calls such as Write-Host and rely on PowerShell's implicit output behavior instead.[1]
powershell.exe -NoProfile -c "Add-Type -AssemblyName System.Web;[System.Web.HttpUtility]::UrlDecode('%C4%9B%C5%A1%C4%8D%C5%99%C5%BE%C3%BD%C3%A1%C3%AD%C3%A9%C5%AF%C3%BA')"

[1] Optional reading: How do I output machine-parseable data from a PowerShell CLI call?

Note: The sample commands are assumed to be run from cmd.exe / outside PowerShell.

  • PowerShell's CLI only supports text as output, not also raw byte data.

  • In order to output data for later programmatic processing, you may have to explicitly ensure that what is output is machine-parseable rather than something that is meant for display only.

There are two basic choices:

  • Rely on PowerShell's default output formatting for outputting what are strings (text) to begin with, as well as for numbers - though for fractional and very large or small non-integer numbers additional effort may be required.

  • Explicitly use a structured, text-based data format, such as CSV or Json, to represent complex objects.


Rely on PowerShell's default output formatting:

  • If the output data is itself text (strings), no extra effort is needed. This applies to your case, and therefore simply implicitly outputting the string returned from the [System.Web.HttpUtility]::UrlDecode() call is sufficient:

    # A simple example that outputs a 512-character string; note that 
    # printing to the _console_ (terminal) will insert line breaks for 
    # *display*, but the string data itself does _not_ contain any 
    # (other than a single _trailing one), irrespective of the 
    # console window width:
    powershell -noprofile -c "'x' * 512"
    
  • If the output data comprises numbers, you may have to apply explicit, culture-invariant formatting if your code must run with different cultures in effect:

    • True integer types do not require special handling, as their string representation is in effect culture-neutral.

    • However, fractional numbers ([double], [decimal]) do use the current culture's decimal mark, which later processing may not expect:

      # With, say, culture fr-FR (French) in effect, this outputs
      # "1,2", i.e. uses a *comma* as the decimal mark.
      powershell -noprofile -c "1.2"
      
      # Simple workaround: Let PowerShell *stringify* the number explicitly
      # in an *expandable* (interpolating) string, which uses
      # the *invariant culture* for formatting, where the decimal
      # mark is *always "." (dot).
      # The following outputs "1.2", irrespective of what culture is in effect.    
      powershell -noprofile -c " $num=1.2; \"$num\" "
      
    • Finally, very large and very small [double] values can result in exponential notation being output (e.g., 5.1E-07 for 0.00000051); to avoid that, explicit number formatting is required, which can be done via the .ToString() method:

      # The following outputs 0.000051" in all cultures, as intended.
      powershell -noprofile -c "$n=0.000051; $n.ToString('F6', [cultureinfo]::InvariantCulture)"
      
  • More work is needed if you want to output representations of complex objects in machine-parseable form, as discussed in the next section.

    • Relying on PowerShell's default output formatting is not an option in this case, because implicit output and (equivalent explicit Write-Output calls) cause the CLI to apply for-display-only formatting, which is meaningful to the human observer but cannot be robustly parsed.

      # Produces output helpful to the *human observer*, but isn't
      # designed for *parsing*.
      # `Get-ChildItem` outputs [System.IO.FileSystemInfo] objects.
      powershell -noprofile -c "Get-ChildItem /"
      
    • Note that use of Write-Host is not an alternative: Write-Host fundamentally isn't designed for data output, and the textual representation it creates for complex objects are typically not even meaningful to the human observer - see this answer for more information.


Use a structured, text-based data format, such as CSV or Json:

Note:

  • Hypothetically, the simplest approach is to use the CLI's -OutputFormat Xml option, which serializes the output using the XML-based CLIXML format PowerShell itself uses for remoting and background jobs - see this answer.

  • However, this format is only natively understood by PowerShell itself, and for third-party applications to parse it they'd have to be .NET-based and use the PowerShell SDK.

    • Also, this format is automatically used for both serialization and deserialization if you call another PowerShell instance from PowerShell, with the command specified as a script block ({ ... }) - see this answer. However, there is rarely a need to call the PowerShell CLI from PowerShell itself, and direct invocation of PowerShell code and scripts provides full type fidelity as well as better performance.
  • Finally, note that all serialization formats, including CSV and JSON discussed below, have limits with respect to faithfully representing all aspects of the data, though -OutputFormat Xml comes closest.

PowerShell comes with cmdlets such as ConvertTo-Csv and ConvertTo-Json, which make it easy to convert output to the structured CSV and JSON formats.

Using a Get-Item call to get information about PowerShell's installation directory ($PSHOME) as an example; Get-Item outputs a System.IO.DirectoryInfo instance in this case:

C:\>powershell -noprofile -c "Get-Item $PSHOME | ConvertTo-Csv -NoTypeInformation"
"PSPath","PSParentPath","PSChildName","PSDrive","PSProvider","PSIsContainer","Mode","BaseName","Target","LinkType","Name","FullName","Parent","Exists","Root","Extension","CreationTime","CreationTimeUtc","LastAccessTime","LastAccessTimeUtc","LastWriteTime","LastWriteTimeUtc","Attributes"
"Microsoft.PowerShell.Core\FileSystem::C:\Windows\System32\WindowsPowerShell\v1.0","Microsoft.PowerShell.Core\FileSystem::C:\Windows\System32\WindowsPowerShell","v1.0","C","Microsoft.PowerShell.Core\FileSystem","True","d-----","v1.0","System.Collections.Generic.List`1[System.String]",,"v1.0","C:\Windows\System32\WindowsPowerShell\v1.0","WindowsPowerShell","True","C:\",".0","12/7/2019 4:14:52 AM","12/7/2019 9:14:52 AM","3/14/2021 10:33:10 AM","3/14/2021 2:33:10 PM","11/6/2020 3:52:41 AM","11/6/2020 8:52:41 AM","Directory"

Note: -NoTypeInformation is no longer needed in PowerShell (Core) 7+

C:\>powershell -noprofile -c "Get-Item $PSHOME | ConvertTo-Json -Depth 1"
{
    "Name":  "v1.0",
    "FullName":  "C:\\Windows\\System32\\WindowsPowerShell\\v1.0",
    "Parent":  {
                   "Name":  "WindowsPowerShell",
                   "FullName":  "C:\\Windows\\System32\\WindowsPowerShell",
                   "Parent":  "System32",
                   "Exists":  true,
                   "Root":  "C:\\",
                   "Extension":  "",
                   "CreationTime":  "\/Date(1575710092565)\/",
                   "CreationTimeUtc":  "\/Date(1575710092565)\/",
                   "LastAccessTime":  "\/Date(1615733476841)\/",
                   "LastAccessTimeUtc":  "\/Date(1615733476841)\/",
                   "LastWriteTime":  "\/Date(1575710092565)\/",
                   "LastWriteTimeUtc":  "\/Date(1575710092565)\/",
                   "Attributes":  16
               },
    "Exists":  true
    // ...
}

Since JSON is a hierarchical data format, the serialization depth must be limited with -Depth in order to prevent "runaway" serialization when serializing arbitrary .NET types; this isn't necessary for [pscustomobject] and [hashtable] object graphs composed of primitive .NET types only.

mklement0
  • 382,024
  • 64
  • 607
  • 775
  • Web server pass POST received data to the STDIN of cmd script, so no need percent doubling in this case. `-NoProfile` cut execution time by half to 1.2 secs is far enough for me, thanks. `Write-Host` usage is TRUE necessary in PS oneliners, otherwise PS console cut STDOUT https://ss64.com/ps/write-host.html – user2956477 Mar 14 '21 at 09:11
  • @user2956477, understood re stdin; I've updated the answer to make it clearer that escaping is only needed for _literals_. Glad to hear that `-NoProfile` helped. `Write-Host` is actually _not_ needed with _string_ output, and is in general always the wrong tool for outputting _data_ - please see the bottom section I've added to the answer. – mklement0 Mar 14 '21 at 15:02
  • Thanks for your addendum, hopefully I understan it even partialy. Unsing many PS various oneliners in my batch script, so should I replace all trailing Write-Host with Write-Output? Does this reliable avoid unwanted 'console width -1' linewraps? – user2956477 Mar 14 '21 at 20:59
  • @user2956477, you don't need explicit `Write-Output` calls: any result that isn't captured in a variable, sent to another command in a (PowerShell command-internal) pipeline or redirected to a file is _implicitly_ output, as shown in the answer. If you have a console width problem, the implication is that you're mistakenly outputting _for-display-only_ representations of _complex objects_, and in that case you need to resort to explicit use of structured data formats instead - or explicitly choose to output just _one property_ of such objects that is itself either a string or a number. – mklement0 Mar 14 '21 at 21:14
  • Thanks a lot (no irony) you tried to teach me internal PS "output producing" issues. I do not know PS at all, my only 'language' are batch scripts (I am not a programmer). Utilizing PS oneliners founded on internet for tasks out of batch scripts ability. I am not able to discovery which oneliner produce what type of output. I need practical way to ensure the textual output will not be distorted, and Write-Host give me this. Maybe this is not correct way, but it enough and I have no good reason to change it. – user2956477 Mar 14 '21 at 21:59
  • Glad to hear it, @user2956477. I understand the pragmatic perspective, but to give you two hints: (a) your command should work _without_ `Write-Host` and (b) here's an example of where `Write-Host` is unhelpful: `powershell -NoProfile -c "@{ foo = 1; bar = 2 } | Write-Host"`. To put it differently: you need to understand _what types of objects_ the PowerShell CLI call outputs and plan accordingly. – mklement0 Mar 14 '21 at 22:15