Decreased output with PowerShell multithreading than with singlethread script

Question

I am using PowerShell 2.0 on a Windows 7 desktop. I am attempting to search the enterprise CIFS shares for keywords/regex. I already have a simple single threaded script that will do this but a single keyword takes 19-22 hours. I have created a multithreaded script, first effort at multithreading, based on the article by Surly Admin.

Can Powershell Run Commands in Parallel?

Powershell Throttle Multi thread jobs via job completion

and the links related to those posts.

I decided to use runspaces rather than background jobs as the prevailing wisdom says this is more efficient. Problem is, is I am only getting partial resultant output with the multithreaded script I have. Not sure if it is an I/O thing or a memory thing, or something else. Hopefully someone here can help. Here is the code.

cls
Get-Date
Remove-Item C:\Users\user\Desktop\results.txt

$Throttle = 5 #threads

$ScriptBlock = {
    Param (
        $File
    )
    $KeywordInfo = Select-String -pattern KEYWORD -AllMatches -InputObject $File
    $KeywordOut = New-Object PSObject -Property @{
        Matches = $KeywordInfo.Matches
        Path = $KeywordInfo.Path
    }
    Return $KeywordOut
}

$RunspacePool = [RunspaceFactory]::CreateRunspacePool(1, $Throttle)
$RunspacePool.Open()
$Jobs = @()

$Files = Get-ChildItem -recurse -erroraction silentlycontinue
ForEach ($File in $Files) {
    $Job = [powershell]::Create().AddScript($ScriptBlock).AddArgument($File)
    $Job.RunspacePool = $RunspacePool
    $Jobs += New-Object PSObject -Property @{
        File = $File
        Pipe = $Job
        Result = $Job.BeginInvoke()
    }
}

Write-Host "Waiting.." -NoNewline
Do {
    Write-Host "." -NoNewline
    Start-Sleep -Seconds 1
} While ( $Jobs.Result.IsCompleted -contains $false)
Write-Host "All jobs completed!"

$Results = @()
ForEach ($Job in $Jobs) {
    $Results += $Job.Pipe.EndInvoke($Job.Result)
    $Job.Pipe.EndInvoke($Job.Result) | Where {$_.Path} | Format-List | Out-File -FilePath C:\Users\user\Desktop\results.txt -Append -Encoding UTF8 -Width 512
}

Invoke-Item C:\Users\user\Desktop\results.txt
Get-Date

This is the single threaded version I am using that works, including the regex I am using for socials.

cls
Get-Date

Remove-Item C:\Users\user\Desktop\results.txt

$files = Get-ChildItem -recurse -erroraction silentlycontinue

ForEach ($file in $files) {
    Select-String -pattern '[sS][sS][nN]:*\s*\d{3}-*\d{2}-*\d{4}' -AllMatches -InputObject $file | Select-Object matches, path |
        Format-List | Out-File -FilePath C:\Users\user\Desktop\results.tx -Append -Encoding UTF8 -Width 512
}

Get-Date
Invoke-Item C:\Users\user\Desktop\results.txt

Answer 1

I am hoping to build this answer over time as I dont want to over comment. I dont know yet why you are losing data from the multithreading but i think we can increase performace with an updated regex. For starters you have many greedy quantifiers that i think we can shrink down.

[sS][sS][nN]:*\s*\d{3}-*\d{2}-*\d{4}

Select-String is case insensitive by default so you dont need the portion in the beginning. Do you have to check for multiple colons? Since you looking for 0 or many :. Same goes for the hyphens. Perhaps these would be better with ? which matches 0 or 1.

ssn:?\s*\d{3}-?\d{2}-?\d{4}

This is assuming you are looking for mostly proper formatted SSN's. If people are hiding them in text maybe you need to look for other delimiters as well.

I would also suggest adding the text to separate files and maybe combining them after execution. If nothing else just to test.

Hoping this will be the start of a proper solution.

Answer 2

It turns out that for some reason the Select-String cmdlet was having problems with the multithreading. I don't have enough of a developer background to be able to tell what is happening under the hood. However I did discover that by using the -quiet option in Select-String, which turns it into a boolean output, I was able to get the results I wanted.

The first pattern match in each document gives a true value. When I get a true then I return the Path of the document to an array. When that is finished I run the pattern match against the paths that were output from the scriptblock. This is not quite as effective performance wise as I had hoped for but still a pretty dramatic improvement over singlethread.

The other issue I ran into was the read/writes to disk by trying to output results to a document at each stage. I have changed that to arrays. While still memory intensive, it is much quicker.

Here is the resulting code. Any additional tips on performance improvement are appreciated:

cls
Remove-Item C:\Users\user\Desktop\output.txt

$Throttle = 5 #threads

$ScriptBlock = {
   Param (
      $File
   )
   $Match = Select-String -pattern 'ssn:?\s*\d{3}-?\d{2}-?\d{4}' -Quiet -InputObject $File
   if ( $Match -eq $true ) {
        $MatchObjects = Select-Object -InputObject $File
        $MatchOut = New-Object PSObject -Property @{
            Path = $MatchObjects.FullName
        }
   }
   Return $MatchOut
}

$RunspacePool = [RunspaceFactory]::CreateRunspacePool(1, $Throttle)
$RunspacePool.Open()
$Jobs = @()

$Files = Get-ChildItem -Path I:\ -recurse -erroraction silentlycontinue
ForEach ($File in $Files) {
   $Job = [powershell]::Create().AddScript($ScriptBlock).AddArgument($File)
   $Job.RunspacePool = $RunspacePool
   $Jobs += New-Object PSObject -Property @{
      File = $File
      Pipe = $Job
      Result = $Job.BeginInvoke()
   }
}

$Results = @()
ForEach ($Job in $Jobs) {
    $Results += $Job.Pipe.EndInvoke($Job.Result)
}

$PathValue = @()
ForEach ($Line in $Results) {
    $PathValue += $Line.psobject.properties | % {$_.Value}
}

$UniqValues = $PathValue  | sort | Get-Unique

$Output = ForEach ( $Path in $UniqValues ) {
    Select-String -Pattern '\d{3}-?\d{2}-?\d{4}' -AllMatches -Path $Path | Select-Object -Property Matches, Path
}

$Output | Out-File -FilePath C:\Users\user\Desktop\output.txt -Append -Encoding UTF8 -Width 512

Invoke-Item C:\Users\user\Desktop\output.txt