Running tasks parallel in powershell

Question

I have a PowerShell script like this:

Foreach ($file in $files) {
    [Do something]
    [Do something]
    [Do something]
}

This way one file is treated after the other. I want to treat 4 files at the same time.

I know of the foreach -parallel loop, but that does the [do something] tasks in parallel. I basically want to run the whole foreach loop in parallel.

How can I achieve this in PowerShell?

score 42 · Accepted Answer · edited May 23 '17 at 10:31

42

You might look into Jobs or runspaces. Here is an example of Jobs:

$block = {
    Param([string] $file)
    "[Do something]"
}
#Remove all jobs
Get-Job | Remove-Job
$MaxThreads = 4
#Start the jobs. Max 4 jobs running simultaneously.
foreach($file in $files){
    While ($(Get-Job -state running).count -ge $MaxThreads){
        Start-Sleep -Milliseconds 3
    }
    Start-Job -Scriptblock $Block -ArgumentList $file
}
#Wait for all jobs to finish.
While ($(Get-Job -State Running).count -gt 0){
    start-sleep 1
}
#Get information from each job.
foreach($job in Get-Job){
    $info= Receive-Job -Id ($job.Id)
}
#Remove all jobs created.
Get-Job | Remove-Job

In the above code I have it where each $file is running in parallel with eachother (Up to 4 running simultaneously).

EDIT: In response to the comments, here is some documentation about scriptblocks. The short reason about why you must include the parameter is because unlike PowerShell functions, scriptblocks can't specify parameters outside of the braces {}.

edited May 23 '17 at 10:31

Community

1
1

answered Apr 28 '17 at 17:39

CuriousOne

922
1
10
26

5

Nicely done, though I would recommend using the `Wait-Job` cmdlet instead of all the fancy logic you include to get running jobs and wait for them to complete. [Link to alternative walk-through of runspaces that I personally learned it from and like.](https://foxdeploy.com/2016/05/17/part-v-powershell-guis-responsive-apps-with-progress-bars/) – TheMadTechnician Apr 28 '17 at 17:43
Oh, and you should probably explain why you have to pass the `$file` parameter into your scriptblock. – TheMadTechnician Apr 28 '17 at 17:44
I'll look into that for future scripts. I had no idea `Wait-Job` existed. – CuriousOne Apr 28 '17 at 17:45
1

This method of using jobs is way slower for loading and filtering files. I thought it would be faster because 4 files could be loaded and filtered at the same time. But actually loading and filtering one file after another is a lot faster. How can you explain this? And is there a way to make it faster? – Werner Schoemaker Apr 29 '17 at 09:39
2

Would you be willing to add an example using runspaces? I am reading up and trying to convert your Jobs example to use runspaces but can't figure it out. – IMTheNachoMan Feb 10 '18 at 18:26

score 13 · Answer 2 · answered Jan 02 '20 at 08:48

Powershell 7 introduces foreach-object -parallel:

https://devblogs.microsoft.com/powershell/powershell-foreach-object-parallel-feature/

Your script would then say

$files | ForEach-Object -parallel {
    [Do something]
    [Do something]
    [Do something]
}

score 5 · Answer 3 · answered Apr 28 '17 at 17:32

5

According to Get-Help about_Foreach-Parallel, ForEach -Parallel... will process the entire scriptblock in parallel for each item, but the commands in the scriptblock will be processed sequentially (though presumably they will be parallelized if bracketed with Parallel {...}). However, your script must be a PowerShell workflow for this to be accepted; the Parallel and Sequence keywords are only effective within workflows.

answered Apr 28 '17 at 17:32

Jeff Zeitlin

9,773
2
21
33

7

From https://msdn.microsoft.com/en-us/powershell/reference/5.1/psworkflow/about/about_foreach-parallel : Describes the ForEach -Parallel language construct in Windows PowerShell Workflow. The -Parallel parameter doesn't exist if you're *not* using a workflow. I didn't get the feeling the OP is working with workflows. – tommymaynard Apr 28 '17 at 17:54
@tommymaynard - Yes, the `Parallel` switch and construct are only applicable to workflows, and I did note that. The querent did mention the `Foreach -parallel` construct in his question, but appeared to misunderstand the effect, so I assumed that a workflow was a possibility, and answered on that basis. If he's not, the job solution that Curious One gave might be the way to go. – Jeff Zeitlin Apr 28 '17 at 17:58
You're absolutely right, you did mention workflows. Sorry about that! – tommymaynard Apr 28 '17 at 20:54

Running tasks parallel in powershell

3 Answers3

Linked