Ways to detect an external program in the pipeline has exited

Question

In a pipeline like

GenerateLotsOfText | external-program | ConsumeExternalProgramOutput

when external-program exits, the pipeline just keeps on running until GenerateLotsOfText completes. Suppose external-program generates only one line of output then

GenerateLotsOfText | external-program | Select-Object -First 1 | ConsumeExternalProgramOutput

will stop the whole pipeline from the moment external-program generates output. This is the behavior I'm looking for, but the caveat is that when external-program generates no output but exits prematurely (because of ctrl-c for instance) the pipe still keeps on running. So I'm now looking for a nice way to detect when that happens and have the pipe terminate when it does.

It seems it's possible to write a cmdlet that uses System.Diagnostics.Process and then use Register-ObjectEvent to listen for the 'Exit' event, but that's quite involved with the handling of I/O streams, etc so I'd rather find another way.

I figured pretty much all other shells have this 'produce output when program exits' builtin via || and && and indeed this works:

GenerateLotsOfText | cmd /c '(external-program && echo FOO) || echo FOO' | Select-Object -First 1 | FilterOutFooString | ConsumeExternalProgramOutput

so no matter what external-program does, a line FOO will always be produced when it exits so the pipe will stop immediately (and then the FilterOutFooString takes care of only producing actual output). This isn't particularly 'nice' and has extra overhead because everything needs piping through cmd (or any other shell would work as well I assume). I was hoping pipeline chain operators would allow this natively but they don't seem to: trying the same syntax results in Expressions are only allowed as the first element of a pipeline. The chaining does work as expected, just not in the pipeline e.g. this yields the aforementioned ParserError:

GenerateLotsOfText | ((external-program && echo FOO) || echo FOO) | Select-Object -First 1

Is there another way to accomplish this?

Update other possible approaches:

• use a separate runspace that polls $LASTEXITCODE in the runspace which runs the external command. Didn't find a way to do that in a thread-safe way though (e.g. $otherRunspace.SessionStateProxy.GetVariable('LASTEXITCODE') cannot be called from another thread when the pipeline is running)
• same idea but for something else to poll on: in the NativeCommandProcessor can be seen that it will set the ExecutionFailed failed flag on the parent pipeline once the external process exits, but I didn't find a way to access that flag, let alone in a thread-safe way
• I might be onto something using a SteppablePipeline. If I get it correctly it gets an object which allows manually executing pipeline iterations. Which would allow checking $LASTEXITCODE in between iterations.

looking into this after having implemented the last idea which is really straightforward, none of the above are an option: the process exit code basically only gets determined once the end block of the pipeline runs, i.e. after the upstream pipeline element produced all its output

Answer 1

To be clear: As originally reported for Unix and then for Windows, PowerShell's current behavior (as of v7.2) should be considered a bug that will hopefully be fixed: PowerShell should detect when an native (external program) has exited and should stop all upstream commands in that event.

---

The following shows a generic workaround, which, however, invariably slows things down and has a limitation:

• As of v7.2, PowerShell unfortunately doesn't offer a public feature to stop upstream commands (see GitHub feature request #); internally, it uses a private exception, as used by Select-Object -First, for instance. The function below uses reflection and ad-hoc-compiled C# code to throw this exception (as also demonstrated in this answer). This means that a performance penalty for the compilation is paid on first invocation of the function in a session.

Define the function nw ("native wrapper"), whose source code is below, and then invoke it as follows, for instance:

• On Windows (assumes WSL; omit -Verbose to suppress verbose output):

  1..1e8 | nw wsl head -n 10 -Verbose
  

• On Unix-like platforms (omit -Verbose to suppress verbose output):

  1..1e8 | nw head -n 10 -Verbose
  

You'll see that the upstream command - the enumeration of the large input array - is terminated when head terminates (for the reasons stated, this termination of the upstream command incurs a performance penalty on first invocation in a session).

Implementation note:

• nw is implemented as a proxy (wrapper) function, using a steppable pipeline - see this answer for more information.

function nw {
  [CmdletBinding(PositionalBinding = $false)]
  param(
    [Parameter(Mandatory, ValueFromRemainingArguments)]
    [string[]] $ExeAndArgs
    ,
    [Parameter(ValueFromPipeline)]
    $InputObject
  )
  
  begin {

    # Split the arguments into executable name and its arguments.
    $exe, $exeArgs = $ExeAndArgs

    # Determine the process name matching the executable.
    $exeProcessName = [IO.Path]::GetFileNameWithoutExtension($exe)

    # The script block to use with the steppable pipeline.
    # Simply invoke the external program, and PowerShell will pipe input
    # to it when `.Process($_)` is called on the steppable pipeline in the `process` block,
    # including automatic formatting (implicit Out-String -Stream) for non-string input objects.
    # Also, $LASTEXITCODE is set as usual.
    $scriptCmd = { & $exe $exeArgs }

    # Create the steppable pipeline.
    try {
      $steppablePipeline = $scriptCmd.GetSteppablePipeline($myInvocation.CommandOrigin)
      # Record the time just before the external process is created.
      $beforeProcessCreation = [datetime]::Now
      # Start the pipeline, which creates the process for the external program
      # (launches it) - but note that when this call returns the
      # the process is *not* guaranteed to exist yet.
      $steppablePipeline.Begin($PSCmdlet)
    }
    catch {
      throw # Launching the external program failed.
    }
    # Get a reference to the newly launched process by its name and start time.
    # Note: This isn't foolproof, but probably good enough in practice.
    # Ideally, we'd also filter by having the current process as the parent process,
    # but that would require extra effort (a Get-CimInstance call on Windows, a `ps` call on Unix)
    $i = 0
    while (-not ($ps = (Get-Process -ErrorAction Ignore $exeProcessName | Where-Object StartTime -GT $beforeProcessCreation | Select-Object -First 1))) {
      if (++$i -eq 61) { throw "A (new) process named '$exeProcessName' unexpectedly did not appear with in the timeout period or exited right away." }
      Start-Sleep -Milliseconds 50
    }
  }
  
  process {
    
    # Check if the process has exited prematurely.
    if ($ps.HasExited) {

      # Note: $ps.ExitCode isn't available yet (even $ps.WaitForExit() wouldn't help), 
      #       so we cannot use it in the verbose message.
      #       However, $LASTEXITCODE should be set correctly afterwards.
      Write-Verbose "Process '$exeProcessName' has exited prematurely; terminating upstream commands (performance penalty on first call in a session)..."
      
      $steppablePipeline.End()

      # Throw the private exception that stops the upstream pipeline
      # !! Even though the exception type can be obtained and instantiated in
      # !! PowerShell code as follows:
      # !!   [System.Activator]::CreateInstance([psobject].assembly.GetType('System.Management.Automation.StopUpstreamCommandsException'), $PSCmdlet)
      # !! it cannot be *thrown* in a manner that *doesn't* 
      # !! result in a *PowerShell error*. Hence, unfortunately, ad-hoc compilation
      # !! of C# code is required, which incurs a performance penalty on first call in a given session.
    (Add-Type -PassThru -TypeDefinition '
      using System.Management.Automation;
      namespace net.same2u.PowerShell {
        public static class CustomPipelineStopper {
          public static void Stop(Cmdlet cmdlet) {
            throw (System.Exception) System.Activator.CreateInstance(typeof(Cmdlet).Assembly.GetType("System.Management.Automation.StopUpstreamCommandsException"), cmdlet);
          }
        }
    }')::Stop($PSCmdlet)
    
    }

    # Pass the current pipeline input object to the target process
    # via the steppable pipeline.
    $steppablePipeline.Process($_)

  }
  
  end {
    $steppablePipeline.End()
  }
}