1

Simple question, but its solution will become life-Saver for me.

I have powershell script as below that just copy paste some source folders to/on some destination locations as routeen as weekly backup.

Source folders are on windows Server2K12 on HDD with multi core processor & destinations on directly connected removable SSD as backup drive.

Window's copy-paste is very painfull as stuck inbetween operation, & mostly stop incomplete.

Through Powershell , can get job done even on any file/folder attribute with it's powerful -force - recurse parameters.

But i can't integrate that 4 script blocks , which require to run in parallel ,so, in loop with Start-Rs job. (I choose RSjob, because in POwershell-ISE there are no any Foreach-Parallel cmdlet available, its available only in Visual Studio Code & in PWSH)

Code:-

 
 $THROTT=[INT]$env:NUMBER_OF_PROCESSORS + 1
  
 Start-RSJob -ScriptBlock {COPY-ITEM -LiteralPath '\\?\D:\S1' -Destination '\\?\F:\200223\D1\' -Force -Recurse} -Throttle $THROTT 
 Start-RSJob -ScriptBlock {COPY-ITEM -LiteralPath '\\?\D:\S2' -Destination '\\?\F:\200223\D2\' -Force -Recurse} -Throttle $THROTT 
 Start-RSJob -ScriptBlock {COPY-ITEM -LiteralPath '\\?\D:\S3' -Destination '\\?\F:\200223\D3\' -Force -Recurse} -Throttle $THROTT
 Start-RSJob -ScriptBlock {COPY-ITEM -LiteralPath '\\?\D:\S4' -Destination '\\?\F:\200223\D4\' -Force -Recurse} -Throttle $THROTT

GET-RSJOB |FOREACH {WHILE ($_.State -eq 'RUNNING') {START-SLEEP 1 ;Write-Host $($_.NAME) ' ' 'RUNNING'}}

I already study this link powershell run script in parallel but its different, it is about how to take sources/targets for jobs from other source , like from file as via use of Get-Content cmdlet. & It's Great feature Parallel Processing. To dramatically reduce processing time as per Link here.

https://devblogs.microsoft.com/powershell/powershell-foreach-object-parallel-feature/

But my current requirement is direct hard code sources & destinations in current script as they (mostly destinations) often change periodically & I Can not integrate that approach (Sources/Destinations written in/from file) in my current requirement.

My current script try to check each job's status , but result display only for first job only, not for all-job's.

If i can construct all jobs in single loop, then i can check each job's status in loop after some time interval like on every 30 seconds.

I already try construct each block in variable like

$job1 = script-block {...} $job2=script-block {...} etc. But can't success that integrate in loop.

I use powershell-ISE , If, there are solution found, then how to run that script with/through PWSH (Powershell-Console)?

Can I use pwsh's foreach-parallel?, which changes require change in script to run through pwsh?

My sources folders are mostly constants , so i can directly provide only required destinations to script without open , write, & save script for writing changing destinations each time.

Thank you.

Chirag Raval
  • 13
  • 7
  • Unless the source and destination directories are, say, mount points or otherwise scaled out, I don't think parallel copying is going to speed up the process. – vonPryz Apr 27 '23 at 07:58
  • @vonPryz, i used -LiteralPath '\\?\D:\S1 & -Destination '\\?\F:\200223\D1\' just for outcome from limitation of possible Long path, (260 char) as per this link. https://c-nergy.be/blog/?p=15339 – Chirag Raval Apr 27 '23 at 17:45
  • https://stackoverflow.com/a/74257557/15339544 this module handles parallel pipeline processing in pwsh 5.1 will be probably easier than what you currently have – Santiago Squarzon Apr 27 '23 at 18:18

1 Answers1

0

If you're willing to give the function of this answer a try or, preferably, the one in the GitHub repo which has a different code-base and added features, this is how the code would look. It already handles pipeline processing, similar to ForEach-Object -Parallel but compatible with Windows PowerShell 5.1.

@{ LiteralPath = '\\?\D:\S1'; Destination = '\\?\F:\200223\D1\' },
@{ LiteralPath = '\\?\D:\S2'; Destination = '\\?\F:\200223\D2\' },
@{ LiteralPath = '\\?\D:\S3'; Destination = '\\?\F:\200223\D3\' },
@{ LiteralPath = '\\?\D:\S4'; Destination = '\\?\F:\200223\D4\' } |
    Invoke-Parallel { Copy-Item @_ -Force -Recurse } -ThrottleLimit ($env:NUMBER_OF_PROCESSORS + 1)

If you want to install the repo version, you can get it through the PowerShell Gallery:

Install-Module -Name PSParallelPipeline -Scope CurrentUser
Santiago Squarzon
  • 41,465
  • 5
  • 14
  • 37
  • Thank you "https://stackoverflow.com/users/15339544/santiago-squarzon" for your decent solution, I already test it, but Its throw an Error on -ThrottleLimit "-"Invoke-Parallel : A parameter cannot be found that matches parameter name 'ThrottleLimit'. At line:4 char:55 + ... oke-Parallel { Copy-Item @_ -Force -Recurse } -ThrottleLimit ($env:N ...+ _ + CategoryInfo: InvalidArgument: (:) [Invoke-Parallel], Paramete rBindingException + FullyQualifiedErrorId : NamedParameterNotFound,Invoke-Parallel" so i must remove that portion to work – Chirag Raval Apr 28 '23 at 12:15
  • if i remove -throttlelimit parameter, it's working on my local pc, , & if try with apply it, it's throw error!!. Plase guide if , ignored, can it effect on behaviour of your function? – Chirag Raval Apr 28 '23 at 19:36
  • @ChiragRaval the function has a `-ThrottleLimit` parameter you can clearly see it in the code from the linked answer. I dont know what could be wrong with what you're doing. – Santiago Squarzon Apr 28 '23 at 19:37
  • @Santiagosquarzone, parameter removed means not from your function, but ,not used, just ignore when really using /applying your function on production. Is this ignorance, can effect the result of process? – Chirag Raval Apr 28 '23 at 19:46
  • im not sure what you mean @ChiragRaval sorry, the `-ThrottleLimit` parameter is there in the module, see: https://github.com/santisq/PSParallelPipeline/blob/main/PSParallelPipeline/PSParallelPipeline.psm1#L280-L282. I would recommend you to install the module from the gallery using `Install-Module` as shown in my answer. In any case, by default, the function does 5 threads at a time. – Santiago Squarzon Apr 28 '23 at 19:50
  • I Run your Examples from your page on this github [link](https://github.com/santisq/PSParallelPipeline) as $message = 'Hello world from {0}' ;0..10 | Invoke-Parallel { $using:message -f [runspace]::DefaultRunspace.InstanceId Start-Sleep 3 } -ThrottleLimit 3 – Chirag Raval Apr 29 '23 at 09:40
  • **But Error** invoke-Parallel : A parameter cannot be found that matches parameter name 'Variables'. At line:6 char:3 + } -Variables @{message=$message} -ThrottleLimit 3 + + CategoryInfo : InvalidArgument: (:) [Invoke-Parallel], ParameterBindingException + FullyQualifiedErrorId : namedParameterNotFound,Invoke-Parallel Can you look in to the matter? – Chirag Raval Apr 29 '23 at 09:44
  • 1
    Its Resolved Now!!, there are already 1 module names "invoke-parallel" installed in my system, i close powersheel ISE, ,remove that whole folder from modules folder, & verify that your module "PSParallelPipeline" installed there ,then open ISE , Run Your All examples, Thats All Sucess!! – Chirag Raval Apr 29 '23 at 13:03
  • @ChiragRaval awrsome – Santiago Squarzon Apr 29 '23 at 15:12
  • Now, Its Working !!, How can i get result of this process? like get-job, Receive-job? Can this script run in powershell core 7.3.4 ?, can i use Foreach-Object -parallel command instead of "Invoke-Parallel"? in POwershell Core? – Chirag Raval May 01 '23 at 12:23