multithread with runspaces instead of foreach cycle

Question

I have a script with a foreach cycle. It has about a dozen functions, each collecting information from remote machines' C$ share (cutting text files, checking file version, etc.)

This is however taking some time, since each machine's data collected after one by one. (sometimes it runs with 500+ input)

Wish to put this into runspaces with parallel execution, but so far no examples worked. I am quite new to the concept.

Current script's outline $inputfile = c:\temp\computerlist.txt

function 1 function 2 function 3, etc

foreach cycle function 1 function 2 function 3

All results written to screen with write-host for now.

What's your script? I believe invoke-command on a remote computer runs in the background already. — js2010, Jul 08 '19 at 14:19
yea, but invoke-command cannot be used in this environment. I am writing simple functions to check file version, or cut a .log file string, etc... it's all through the C$ share, but one by one in a foreach cycle for now... and for 500+ machines it can take hours — DanZi DanZi Rulez, Jul 09 '19 at 15:06
There are many ways to run things in the background, jobs, start-process, workflows... https://stackoverflow.com/questions/56612034/wait-for-multiple-simultaneous-powershell-commands-in-other-sessions-to-finish-b/56926545#56926545 — js2010, Jul 09 '19 at 15:10

score 0 · Answer 1 · answered Jul 08 '19 at 15:19

This example pings a number of server in parallel, so you easily can modify it for your demands:

Add-Type -AssemblyName System.Collections

$GH = [hashtable]::Synchronized(@{})

[System.Collections.Generic.List[PSObject]]$GH.results = @()
[System.Collections.Generic.List[string]]$GH.servers = @('server1','server2','server3');
[System.Collections.Generic.List[string]]$GH.functions = @('Check-Server');

[System.Collections.Generic.List[PSObject]]$jobs = @()


#-----------------------------------------------------------------
function Check-Server {
#-----------------------------------------------------------------

   # a function which runs parallel

   param(
        [string]$server
    )

    $result = Test-Connection $server -Count 1 -Quiet 
    $GH.results.Add( [PSObject]@{ 'Server' = $server; 'Result' = $result } )

}

#-----------------------------------------------------------------
function Create-InitialSessionState {
#-----------------------------------------------------------------

    param(
        [System.Collections.Generic.List[string]]$functionNameList
    )

    # Setting up an initial session state object
    $initialSessionState = [System.Management.Automation.Runspaces.InitialSessionState]::CreateDefault()

    foreach( $functionName in $functionNameList ) {

        # Getting the function definition for the functions to add
        $functionDefinition = Get-Content function:\$functionName
        $functionEntry = New-Object System.Management.Automation.Runspaces.SessionStateFunctionEntry -ArgumentList $functionName, $functionDefinition

        # And add it to the iss object
        [void]$initialSessionState.Commands.Add($functionEntry)
    }

    return $initialSessionState
}

#-----------------------------------------------------------------
function Create-RunspacePool {
#-----------------------------------------------------------------

    param(
        [InitialSessionState]$initialSessionState
    )

    $runspacePool = [RunspaceFactory]::CreateRunspacePool(1, ([int]$env:NUMBER_OF_PROCESSORS + 1), $initialSessionState, $Host)
    $runspacePool.ApartmentState = 'MTA'
    $runspacePool.ThreadOptions  = "ReuseThread"
    [void]$runspacePool.Open()

    return $runspacePool
}


#-----------------------------------------------------------------
function Release-Runspaces {
#-----------------------------------------------------------------

    $runspaces = Get-Runspace | Where { $_.Id -gt 1 }

    foreach( $runspace in $runspaces ) {
        try{
            [void]$runspace.Close()
            [void]$runspace.Dispose()
            }
        catch {
        }
    }
}


$initialSessionState = Create-InitialSessionState -functionNameList $GH.functions
$runspacePool        = Create-RunspacePool -initialSessionState $initialSessionState


foreach ($server in $GH.servers)
{
    Write-Host $server

    $job = [System.Management.Automation.PowerShell]::Create($initialSessionState)
    $job.RunspacePool = $runspacePool

    $scriptBlock = { param ( [hashtable]$GH, [string]$server ); Check-Server -server $server }

    [void]$job.AddScript( $scriptBlock ).AddArgument( $GH ).AddArgument( $server )
    $jobs += New-Object PSObject -Property @{
                                    RunNum = $jobCounter++
                                    JobObj = $job
                                    Result = $job.BeginInvoke() }

    do {
        Sleep -Seconds 1 
    } while( $runspacePool.GetAvailableRunspaces() -lt 1 )

}

Do {
    Sleep -Seconds 1
} While( $jobs.Result.IsCompleted -contains $false)


$GH.results

Release-Runspaces | Out-Null

[void]$runspacePool.Close()
[void]$runspacePool.Dispose()

Hi, any idea why the script would simply stop when a ping is not successful? I was trying to modify the ping part, where it would only add results to $GH.results when the ping is successful. Either way, it hangs without any error message on the first machine that it cannot ping. — DanZi DanZi Rulez, Jul 22 '19 at 11:30

js2010 · Answer 2 · 2019-07-09T21:03:13.900

This would be concurrent and run in about 10 seconds total. The computer could be localhost three times if you got it working.

invoke-command comp1,comp2,comp3 { sleep 10; 'done' }

Simple attempt at api (threads):

$a =  [PowerShell]::Create().AddScript{sleep 5;'a done'}
$b =  [PowerShell]::Create().AddScript{sleep 5;'b done'}
$c =  [PowerShell]::Create().AddScript{sleep 5;'c done'}
$r1 = $a.BeginInvoke(); $r2 = $b.BeginInvoke() $r3 = $c.BeginInvoke()
$a.EndInvoke($r1); $b.EndInvoke($r2); $c.EndInvoke($r3)

a done
b done
c done

multithread with runspaces instead of foreach cycle

2 Answers2