In a powershell pipeline, how to wait for a cmdlet to complete before proceding to the next one?

Question

Consider the following function:

function myFunction {
  100
  sleep 1
  200
  sleep 1
  300
  sleep 1
}

As you can see it will emit those values, one by one down the pipeline.

But I want to wait for all the values to be emitted before going on. Like

myFunction | waitForThePreviousCommandToComplete | Format-Table 

I want the Format-Table above to receive the entire array, instead of one-by-one items.

Is it even possible in Powershell?

Answer 1

Use (...), the grouping operator in order to collect a command's output in full first, before sending it to the success output stream (the pipeline).

# Due to (...), doesn't send myfunction's output to Format-Table until it has run 
# to completion and all its output has been collected.
(myFunction) | Format-Table

# Also works for entire pipelines.
(100, 200, 300 | ForEach-Object { $_; Start-Sleep 1 }) | Format-Table

Note:

• If you need to up-front collect the output of multiple commands (pipelines) and / or language statements, use $(...), the subexpression operator instead, e.g.
  $(Get-Date -Year 2020; Get-Date -Year 2030) | Format-Table; the next point applies to it as well.

• Whatever output was collected by (...) is enumerated, i.e., if the collected output is an enumerable, its elements are emitted one by one to the success output stream - albeit without any delay at that point.

  • Note that the collected output is invariably an enumerable (an array of type [object[]]) if two or more output objects were collected, but it also can be one in the usual event that a single object that itself is an enumerable was collected.
  • E.g., (Write-Output -NoEnumerate 1, 2, 3) | Measure-Object reports a count of 3, even though Write-Output -NoEnumerate output the given array as a single object (without (...), Measure-Object would report 1).

• Typically, commands (cmdlets, functions, scripts) stream their output objects, i.e. emit them one by one to the pipeline, as soon as they are produced, while the command is still running, as your function does, and also act on their pipeline input one by one. However, some cmdlets invariably, themselves collect all input objects first, before they start emitting their output object(s), of conceptual necessity: notable examples are Sort-Object, Group-Object, and Measure-Object, all of which must act on the entirety of their input before they can start emitting results. Ditto for Format-Table when it is passed the -AutoSize switch, discussed next.

• In the case of Format-Table, specifically, you can use the -AutoSize switch in order force it to collect all input first, in order to determine suitable display column widths based on all data (by default, Format-Table waits for 300 msecs. in order to determine column widths, based on whatever subset of the input data it has received by then).

  • However, this does not apply to so-called out-of-band-formatted objects, notably strings and primitive .NET types, which are still emitted (by their culture-invariant .ToString() representation) as they're being received.

  • Only complex objects (those with properties) are collected first, notably hashtables and [pscustomobject] instances; e.g.:

    # Because this ForEach-Object call outputs complex objects (hashtables),
    # Format-Table, due to -AutoSize, collects them all first,
    # before producing its formatted output.
    100, 200, 300 | ForEach-Object { @{ num = $_ }; Start-Sleep 1 } |
      Format-Table -AutoSize
    

If you want to create a custom function that collects all of its pipeline input up front, you have two options:

• Create a simple function that uses the automatic $input variable in its function body, which implicitly runs only after all input has been received; e.g.:

  # This simple function simply relays its input, but 
  # implicitly only after all of it has been collected.
  function waitForThePreviousCommandToComplete { $input }
  
  # Output doesn't appear until after the ForEach-Object
  # call has emitted all its output.
  100, 200, 300 | ForEach-Object { $_; Start-Sleep 1 } | waitForThePreviousCommandToComplete
  

• In the context of an advanced function, you'll have to manually collect all input, iteratively in the process block, via a list-type instance allocated in the begin block, which you can then process in the end block.

  • While using a simple function with $input is obviously simpler, you may still want an advanced one for all the additional benefits it offers (preventing unbound arguments, parameter validation, multiple pipeline-binding parameters, ...).
  • See this answer for an example.

Answer 2

Sort waits until it has everything.

myFunction | sort-object

Or:

(myFunction) 
$(myfunction1; myFunction2)
myFunction | format-table -autosize
myFunction | more

See also: How to tell PowerShell to wait for each command to end before starting the next?

Answer 3

For some unknown reason, just putting the function inside brackets solved my problem:

(myFunction) | Format-Table