powershell start process within for each loop

Question

I have a for each loop I wish to run in parallel, however I'm not getting a uniform time as to when each iteration kicks off, which is leading to some edgecase timing issues.

As such, I've changed my loop just to be sequential. However, I don't want to wait for the iteration commands to complete.

Each iteration is essentially doing:

Invoke-Pester @{Path= "tests.ps1"; Parameters = @{...}} -Tag 'value' -OutputFile $xmlpath -OutputFormat NUnitXML -EnableExit

I want each iteration to run sequentially (tests expect to be run sequentially) however I don't wish to wait for tests to complete.

What is the best way to ensure the iteration does not wait for the Invoke-Pester command to complete, such that the next iteration kicks off after the previous iteration has initiated? I've tried using Start-Process Invoke-Pester which I think invalidated further code structure.

Thank you

"I'm not getting a uniform time as to when each iteration kicks off" - can you elaborate on this? What are you expecting, and how are you measuring it? — Mathias R. Jessen, Nov 02 '20 at 18:09
@MathiasR.Jessen 0..10 | ForEach-Object -Parallel {Start-Sleep -Seconds ($_ + $_); #invoke pester... #write-host $testnumber $starttime. What I was finding was that the order of thread execution in parallel, with the staggering start code, did not always yeild testcases being started in order. Sometimes one subsequent testcase may run up to a minute behind the previous testcase. I needed testcase1 launched (ie iteration 1) then a few seconds later testcase2 launched, and so on. Most times this worked out, but sometimes parallel thread orders became slightly mixed up, causing test failures. — , Nov 03 '20 at 15:08
Ahh, I see. `ForEach-Object -Parallel` makes absolutely no guarantees as to the underlying scheduling, so if you want them dispatched at uniform intervals and (maybe more importantly) collect the results in order, your best best is by manually implementing orchestration of the tests on top of a `RunspacePool` - or just run them sequentially and live with the wallclock time :) — Mathias R. Jessen, Nov 03 '20 at 15:56
If timing of side effects is not in itself important (ie. it's _just_ about the order of the output) you can get around that easily by giving wrapping every test in an object that holds a order/ranking value: `0..10 | ForEach-Object -Parallel { [pscustomobject]@{Order = $_; TestResult = Invoke-Pester }} |Sort Order` — Mathias R. Jessen, Nov 03 '20 at 15:58
@marsze Thank you marsze, after trying various options, your solution was the best fit - thank you! — , Nov 06 '20 at 14:32

score 0 · Accepted Answer · answered Nov 02 '20 at 18:13

One way for async processing are PowerShell jobs. A very basic example:

$jobs = foreach ($xmlPath in (...)) {
    Start-job { Invoke-Pester -OutputFile $Args[0] ... } -ArgumentList $xmlPath
}

# get the job results later:
$jobs | Receive-Job

However, jobs are very slow. A more performant but also a little more complex way is using background runspaces:

$jobs = foreach ($xmlPath in (...)) {
    $ps = [Powershell]::Create()
    [void]$ps.AddScript({ Invoke-Pester -OutputFile $Args[0] ... })
    [void]$ps.AddArgument($xmlPath)
    @{
        PowerShell = $ps
        AsyncResult = $ps.BeginInvoke()
    }
}

# get results:
foreach ($job in $jobs) {
    $job.PowerShell.EndInvoke($_.AsyncResult)
    $job.PowerShell.Dispose()
}

Don't forget `Start-ThreadJob` (ships with v6+, can be installed for Windows PowerShell v3, v4, v5.1) and the v7+ `ForEach-Object -Parallel` - see https://stackoverflow.com/a/59440248/45375 — mklement0, Nov 02 '20 at 18:51
You can install the threadjobs module in powershell 5.1 as well, even as current user. — js2010, Nov 02 '20 at 20:07

powershell start process within for each loop

1 Answers1