pwsh: How to use `Start-Process -Passthru` AND redirect both stdout and stderr to the same file?

Question

i'm banging my head against the evergreen problem of stdout and stderr redirection.

Small introduction

In powershell, you can do $p = Start-Process -Passthru -NoNewWindow ... to start a windowless process and get a Process object reference in $p.

There are also a couple of arguments, -RedirectStandardOutput <path> and -RedirectStandardError <path> that effectively redirect the process' stdout/stderr to the files with the given <path>s.

So, let's consider a simple program invocation, runnable both in cmd.exe and pwsh.exe, that generates both stdout and stderr: cmd /c "(ver & echo errrrr >&2)".

I want to get a reference to the process, so i do:

$p = Start-Process -PassThru -NoNewWindow cmd -ArgumentList '/c','(ver & echo errrrr >&2)' -RedirectStandardOutput 'c:\tmp\ver.txt' -RedirectStandardError 'c:\tmp\vererr.txt'

which is the same as running

& cmd /c '(ver & echo errrrr >&2)' >c:\tmp\ver.txt 2>c:\tmp\vererr.txt

BUT start-process gets me a Process object that i can store in $p!

In any case both of these create ver.txt and vererr.txt with the corresponding streams' data (ver=stdout, vererr=stderr).

Question

My question is: how can i merge the program's entire output, then redirect it to a file, AND get a Process object reference?

By "merge the entire output" i mean as when using >file 2>&1:

cmd /c "(ver & echo errrrr >&2)" >c:\tmp\ver.txt 2>&1

This creates only the ver.txt file with the invocation's full output.

The simple solution of specifying the same file with start-process, yields an error:

PS > $p = start-process -PassThru cmd -ArgumentList '/c','(ver & echo errrrr >&2)' -RedirectStandardOutput 'c:\tmp\ver.txt' -RedirectStandardError 'c:\tmp\ver.txt'
start-process : This command cannot be run because "RedirectStandardOutput" and "RedirectStandardError" are same. Give different inputs and Run your command again.
At line:1 char:6
+ $p = start-process -PassThru cmd -ArgumentList '/c','(ver & echo errr …
+      ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+ CategoryInfo          : InvalidOperation: (:) [Start-Process], InvalidOperationException
+ FullyQualifiedErrorId : InvalidOperationException,Microsoft.PowerShell.Commands.StartProcessCommand

Is this even possible in powershell? Handmade solutions like these have deadlock and output order problems, meaning stdout and stderr data may appear NOT in order when outputting both of the streams to the same file: this might even be a .NET problem because even in pure C# i haven't found a reliable way to achieve this...

Answer 1

You need to merge the output streams at the source, which requires use of a shell command and the shell's redirection features.

Since your command already calls a shell, cmd.exe, simply use its redirection features in ore- which then obviates the need for using Start-Process' -RedirectStandardOut / -RedirectStandardErr parameters:

# Make cmd /c merge stderr into stdout and capture the 
# combined output in c:\tmp\combined.txt
$p = Start-Process -PassThru -NoNewWindow -FilePath cmd -ArgumentList @(
       '/c',
       '(ver & echo errrrr >&2) > c:\tmp\combined.txt 2>&1'
     )

The alternative is to only use 2>&1 as part of the shell command - which merges stderr into stdout and therefore emits all output via stdout - and combine that with -RedirectStandardOut:

# Make cmd /c merge stderr into stdout and capture the 
# combined output in c:\tmp\combined.txt
$p = Start-Process -PassThru -NoNewWindow `
       -RedirectStandardOut c:\tmp\combined.txt -FilePath cmd -ArgumentList @(
         '/c',
         '(ver & echo errrrr >&2) 2>&1'
       )

Note that the character encoding of the output file is determined by the console's code page, as reflected in the output from chcp, which defaults to the system's active legacy OEM code page.

If you want UTF-8 output, switching to UTF-8 (code page 65001) must be done before making the cmd call, as follows - you may want to save and restore the previous values:

# Must be called *before* Start-Process
[Console]::InputEncoding = [Console]::OutputEncoding = [Text.Utf8Encoding]::new()

For more information, see this answer.

---

Use with non-shell commands:

If your command isn't a shell command, you must choose which shell to call via, and there are two basic choices; however, note that whatever shell you choose, you need to be familiar with its specific syntax requirements.

• cmd.exe (as used above):

  • Pass the target command to cmd /c

  • Append 2>&1 in order to merge stderr into stdout

  • Either precede that with > yourOutputFile.txt, or use -RedirectStandardOutput yourOutputFile.txt, as shown above.

  • Quoting considerations:

    • " chars. that are part of the target command generally do not require escaping, though if the command has embedded, \"-escaped characters, you situationally need to use ^-quoting for cmd.exe metacharacters such as & that happen to be present between embedded \"...\" sequences.

• powershell.exe (or pwsh.exe, if PowerShell (Core) 7+ is installed):

  • Pass the target command to powershell -Command

    • Also specifying -NoProfile, i.e. powershell -NoProfile -Command is preferable for reasons of performance and execution-environment predictability, unless you truly need PowerShell's profiles to be loaded in order for the command to succeed (such as due to $env:PATH additions performed by a profile).

  • Append *>&1 | ForEach-Object ToString in order to merge all streams into stdout and combine that with -RedirectStandardOutput yourOutputFile.txt, or use *> yourOutputFile.txt directly, but note that the output file's character encoding will differ:

    • In the former case, the character encoding implied by the console's active code page is used.

    • In the latter case, it is in effect the default encoding of PowerShell's Out-File cmdlet that applies (for which > is, in effect, an alias), which is UTF-16LE ("Unicode") in Windows PowerShell and BOM-less UTF-8 in PowerShell (Core) 7+.

  • Quoting considerations:

    • Use \" in order to escape " chars. that are part of the target command.

Note:

• cmd.exe has the advantage of starting up noticeably more quickly than a PowerShell CLI.

• Also, cmd.exe's quoting requirements are simpler.

---

A concrete example:

The following uses cmd.exe:

• to execute php -r "fwrite(STDOUT,'out'); fwrite(STDERR,'err');"

• saving the combined stdout + stderr output to out.txt

Since '...' quoting is used on the PowerShell side, the ' characters embedded in the PHP command must be escaped as ''

$p = Start-Process -PassThru -NoNewWindow -FilePath cmd -ArgumentList @(
       '/c',
       'php -r "fwrite(STDOUT,''out''); fwrite(STDERR,''err'');" > out.txt 2>&1'
     )